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ZMTRODDCTION 


A.  PURPOSE 

The  purpose  of  this  thesis  is  to  analyze  Army  weapon 
systems  Operational  Tests  &  Evaluations  (0T6E)  that  have  been 
conducted  at  Fort  Hunter-Liggett,  California.  Many  systems 
have  not  been  fully  configured  or  prepared  for  operational 
effectiveness  and  suitability  testing  under  realistic  field 
conditions.  Even  though  the  operational  test  may  be  viewed  as 
the  culmination  of  the  weapon  systems  development  process, 
many  Program  Managers  have  little  experience  in  this  area  and 
as  a  result,  significant  problems  are  encountered  during  OT&E. 
This  thesis  provides  an  analysis  of  these  problems  and  the 
reasons  they  occurred  along  with  proposed  solutions, 
enhancement  observations,  and  'lessons  learned*  that  should  be 
useful  to  Program  Managers. 

B.  BACKGROUND 

A  major  issue  that  frequently  confronts  Program  Managers 
and  operational  test  centers  is  the  arrival  of  a  system  for 
testing  when  they  are  not  fully  ready.  These  readiness  issues 
have  resulted  in  delayed  or  truncated  testing,  and  unscheduled 
expenditure  of  large  amounts  of  money  to  resolve  problems  with 
'quick  fixes'.  In  the  current  restrictive  budget  environment. 
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these  problems  can  also  result  in  a  weapon  system's 
cancellation. 

Many  of  these  readiness  issues  may  be  quite  simple  to  fix 
or  prevent.  The  many  mistakes  that  are  reported  may  be  due  to 
the  inexperience  of  project  team  members,  since  they  may 
experience  an  operational  test  only  once  in  several  years. 
Due  to  rotation  of  project  personnel,  much  knowledge  may  be 
lort  and  therefore  systems  may  be  subject  to  repeated 
problems. 

C.  THESIS  OBJECTIVE 

The  objective  of  this  thesis  is  to  identify  the  most 
common  problems  and  enhancements  that  have  affected  Army 
systems  undergoing  operational  testing.  This  information  will 
point  out  to  members  of  the  program  Manager's  Office  and  the 
Operational  Test  Centers  the  importance  of  these  areas  and 
encourage  them  to  give  these  areas  more  emphasis.  Ultimately, 
this  information  can  lead  to  a  smoother,  less  costly,  and  more 
accurate  Operational  Test  &  Evaluation  process.  A  'lessons 
learned'  summary  will  then  be  developed  into  a  format  that  can 
be  useful  to  Program  Managers  as  a  guide  or  checklist  prior  to 
the  operational  test  of  their  systems. 

D.  RESEARCH  QUESTIONS 

The  following  primary  research  question  will  be  addressed 
in  this  study:  What  essential  tasks  should  the  PM  complete  to 
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ensure  that  his  program  is  ready  for  Operational  Test  and 
Evaluation? 

Subsidiary  research  questions  are: 

(1)  What  are  the  common  problems  that  Program  Managers 
fail  to  identify  before  operational  testing? 

(2)  What  are  the  common  program  strategy  mistakes  that 
Program  Managers  make  in  preparation  for  operational 
testing? 

(3)  What  are  the  common  logistical  problems  that  arise  in 
operational  testing? 

(4)  How  did  these  failures  affect  the  testing  process? 

(5)  What  can  Program  Managers  do  to  better  prepare  their 
systems  for  operational  testing? 

E.  SCOPE 

The  scope  of  this  research  is  limited  to  programs  that 
we'.e  tested  at  Fort  Hunter- Ligget .  These  include  the  ADATS 
(LOS-F-H)  air  defense  system.  Avenger  (Pedestal  Mounted 
Stinger)  air  defense  system,  OH-58D  (AHIP)  scout  helicopter 
and  the  Apache  (AH- 64)  attack  helicopter.  The  intent  is  to 
evaluate  operational  test  observations  of  problems  and 
enhancements,  and  to  develop  a  set  of  'lessons  learned'  and 
recommendations  from  them.  The  research  will  also  include 
observations  from  experienced  Fort  Hunter- Liggett  test 
personnel . 

F.  METHODOLOGY 

Research  for  this  thesis  consisted  primarily  of  a 
literature  review  of  Operational  and  Force  Development  Test 
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Reports  located  at  the  Fort  Hunter-Ligget  testing  facility. 
General  Accounting  Office  Reports,  Congressional  Subconunittee 
Reports,  Defense  Systems  Management  College  (DSMC)  lessons 
learned,  and  Department  of  Defense  regulations. 

Information  was  also  gathered  through  interviews  with 
members  of  the  Fort  Hunter -Liggett  test  center.  Many  of  these 
personnel  have  experienced  niimerous  operational  tests  and 
possess  valuable  insight  and  first  hand  experience  with 
testing  problems.  These  data  will  be  especially  useful,  since 
after  action  reports  often  have  shortcomings.  Many  times 
these  reports  lack  detail,  or  tend  to  gloss  over  negative 
aspects  of  the  event  in  question. 

This  thesis  will  utilize  a  comparative  analysis  of 
significant  issues  discovered,  it  will  be  used  to  determine 
relative  importance  and  sources  of  the  problems. 

O.  THESIS  OROANIZATIOM 

The  thesis  is  divided  into  five  chapters.  Chapter  II 
provides  a  review  of  the  acquisition  process  and  applicable 
guidelines  within  the  Department  of  Defense.  It  also  provides 
a  background  of  the  test  and  evaluation  process  and  its 
purpose . 

Chapter  III  identifies  and  discusses  problems, 
enhancements,  and  'lessons  leax-ned*  that  have  occurred  with 
each  weapon  system  during  testing  at  Fort  Hunter-Liggett . 
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Chapter  IV  concentrates  and  categorizes  these  observations 
and  enhancements  into  'lessons  learned'  for  application  to 
future  programs. 

Chapter  V  presents  conclusions  and  recommendations,  and  a 
'lessons  learned'  summary  for  utilization  in  future  military 
operational  testing. 

Many  acronyms  and  edsbreviations  are  used  in  this  thesis. 
They  are  defined  and  discussed  in  the  attached  appendices 
which  provide  information  that  should  help  the  reader  gain  a 
better  understanding  of  this  area  of  the  acquisition  process. 
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II.  THE  ACQUISITION  PROCESS 


A.  BACKGROUMD 

The  DoD  acquisition  system  defines  the  process  to  be  used 
to  plan,  design,  develop,  acquire,  maintain,  and  dispose  of 
all  equipment,  facilities,  and  services  in  DoD.  The  system 
has  been  continuously  revised  to  streamline  the  acquisition 
process,  provide  for  formal  risk  analysis,  and  to  reduce  or 
eliminate  costly  changes  later  in  the  production  cycle. 

In  the  early  1970' s,  the  Department  of  Defense  test 
policies  became  more  formalized  and  placed  greater  enphasis  on 
test  and  evaluation  (T&E)  as  a  continuing  function  throughout 
the  acquisition  cycle.  These  policies  stressed  the  use  of  T&E 
to  reduce  acquisition  risk  and  provide  early  and  continuing 
estiiiiates  of  the  system's  operational  effectiveness  and 
operational  suiteUsility.  In  order  to  meet  these  policy 
objectives,  it  is  necessary  to  fully  integrate  appropriate 
test  activities  into  the  overall  development 
process. [Ref.  1] 

The  defense  system  acquisition  process  underwent  revision 
in  1987  in  an  attempt  to  make  it  less  costly,  less  time 
consuming,  and  more  responsive  to  the  needs  of  the  operational 
conBnunlty.  As  it  is  now  structured,  the  defense  system  life 
cycle  consists  of  the  following  five  phases: 
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(1)  Concept  Exploration/Definition 

(2)  Concept  Demonstration/Validation 

(3)  Engineering  and  Manufacturing  Development 

(4)  Full -Rate  Production/Deployment 

(5)  Operational  Support 

These  phases  are  separated  by  key  decision  points 
(milestones)  during  which  a  decision  authority  reviews  a 
program  and  may  terminate  it  or  authorizes  it  to  advance  to 
the  next  stage  in  the  cycle.  T&E  results  and  planned  T&E  in 
the  future  play  an  in^ortant  part  in  this  process  and  are 
rigorously  assessed  as  part  of  the  milestone  review  process. 

B.  THE  ACQUISITION  PHASES 

The  following  paragraphs  describe  the  five  major  milestone 
decision  points  and  five  phases  of  the  acquisition  process. 
They  provide  a  basis  for  understanding  the  mcuiagement  and 
progressive  decision  making  associated  with  program 
maturation. [Ref.  2] 

1.  Concept  Eaqploratlon/Definltion  Phase 

The  first  phase  in  the  acquisition  process  is  concept 
exploration/definition.  It  begins  after  Milestone  0  grants 
concept  studies  approval .  This  phase  starts  with  an 
assessment  of  the  current  or  projected  U.S.  military 
capcQslllty  to  perform  assigned  missions,  called  a  Mission  Area 
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Analysis  (MAA) .  This  assessment  is  conducted  before  a  new 
acquisition  starts.  The  MAA  evaluates  threat,  friendly 
capabilities,  technological  opportunities,  doctrine,  and  new 
defense  interests.  The  primary  objective  is  to  identify 
deficiencies  and  determine  a  more  effective  means  of 
performing  assigned  tasks.  The  MAA  may  result  in 
recommendations  to: 

1.  Initiate  new  acquisition  programs. 

2.  Change  U.S.  and  allied  concepts  and  doctrine. 

3.  Use  existing  military  or  commercial  systems. 

4.  Modify  or  inprove  an  existing  system,  or 

5 .  Enter  into  a  cooperative  research  and  development 

progrcun  with  one  or  more  allied  nations. 

If  the  MAA  results  in  a  recommendation  to  initiate  a 
new  acquisition  progrcun,  a  mission  need  statement  (MNS)  is 
submitted  to  the  Defense  Acquisition  Executive  (DAE)  .  The  MNS 
is  submitted  with  or  before  a  Program  Objectives  Memorandum 
(POM)  submission  in  which  funds  are  requested. 

This  leads  to  the  Milestone  I  decision  which  will  mark 
the  start  of  a  new  acquisition  program  if  the  decision  is  to 
enter  into  the  next  phase.  Milestone  I  establishes  broad 
goals  and  thresholds  in  the  areas  of  program  cost,  schedule, 
and  operational  effectiveness  and  suitability.  These  broad 
guidelines  give  the  Program  Manager  (PM)  flexibility  to 
develop  innovative  and  cost-effective  solutions.  The 
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Milestone  I  decision  is  made  by  the  Defense  Acquisition 
Executive  and  is  documented  in  an  Acquisition  Decision 
Memorandum  (ADM)  . 

2.  Concept  Demonstration/Validation  Phase 

The  next  stage  of  the  process  is  called  the  concept 
demonstration/validation  (Dem/Val)  phase.  During  this  phase, 
the  following  events  take  place: 

1.  The  feasibility  of  coit^eting  alternatives  is 
demonstrated  and  the  most  capadsle  system  for  fulfilling  the 
mission  is  selected. 

2.  Prototype  systems  are  fabricated  to  support  both  design 
development,  and  testing  and  evaluation  to  identify  areas 
of  risk. 

The  program  office  updates  life- cycle  costs,  sends 
cuinual  funding  input  into  the  Planning,  Programming  and 
Budgeting  System  (PPBS) ,  and  prepares  documentation  during 
this  phase  to  assist  in  the  Milestone  II  decision.  The  Test 
and  Evaluation  Master  Plan  (TEMP)  is  updated  and  the 
Integrated  Program  Summary  (IPS)  is  prepared.  The  IPS 
summarizes  the  results  of  the  concept  demonstration/validation 
phase,  identifies  the  program  alternatives,  and  establishes 
esqplicit  goals  and  thresholds  for  program  cost,  schedule,  and 
operational  effectiveness  and  suitcdDility. 
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3.  Engineering  and  Manufacturing  Development  Phase 

The  third  stage  of  the  acquisition  process  is 
Engineering  and  Manufacturing  Development  (EMD) .  This  begins 
after  the  Defense  Acquisition  Executive  makes  the  Milestone  II 
decision  and  documents  it  in  an  ADM  granting  approval  to 
proceed  with  the  EMD  (previously  called  the  full-scale 
development  or  PSD)  phase.  During  this  phase,  the  PM 
completes  system  development  up  to  the  point  where  an  economic 
decision  can  be  made  whether  or  not  to  produce  the  system  in 
quantity.  Before  this  decision  can  be  made  in  the 
affirmative,  the  PM  must  demonstrate  that  all  technical, 
operational,  and  resource  requirement  thresholds  have  been  met 
and  that  adequate  resources  are  available  to  support 
production  and  deployment.  This  is  done  through  the 
completion  of  developmental  testing  and  the  conduct  of 
operational  testing  in  a  realistic  environment  with  extensive 
user  participation. 

In  preparation  for  Milestone  III,  the  IPS  and  the  TEMP 
are  updated  to  describe  any  program  changes  made  since 
Milestone  II  and  to  propose  goal  and  threshold  revisions,  if 
appropriate.  The  Milestone  III  decision  is  made  by  the 
Secretary  of  Defense  and  recorded  in  an  ADM.  It  either 
terminates  the  project,  requests  further  testing,  or  grants 
approval  to  proceed  with  the  full -rate  production  and 
deployment  phase. 
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4.  Production  and  Deployment  Phase 

The  next  phase  is  production  and  deployment.  During 
this  phase,  the  PM  ensures  that  systems  are  produced  and 
deployed  according  to  plans.  Operational  testing  and 
evaluation,  user  training,  and  logistical  support  are  key 
activities  during  the  production  and  deployment  phase.  A 
formal  review  is  scheduled  one  to  two  years  after  initial 
deployment  of  a  system  to  ensure  that  operational  readiness 
and  support  objectives  are  being  met.  The  results  of  this 
review  are  presented  for  consideration  for  the  Milestone  IV 
decision,  which  identifies  actions  and  resources  needed  to 
ensure  that  objectives  are  achieved  and  maintained. 

5.  Operational  Support  Phase 

The  final  stage  is  the  operational  support  phase. 
This  involves  support  of  the  system  in  the  field,  as  it  is 
monitored  for  suited)ility  and  readiness.  It  also  involves 
determinations  of  whether  major  upgrades  are  necessary,  or  if 
deficiencies  warrant  consideration  of  replacement. 

C.  TEST  AMD  EVALUATION 

The  purposes  of  test  and  evaluation  in  a  defense  system's 
development  and  acquisition  program  are  to  determine  the 
feasibility  of  conceptual  approaches,  to  minimize  design  risk, 
to  identify  design  alternatives,  to  compare  and  analyze 
tradeoffs,  and  to  estimate  operational  effectiveness  and 
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suitability.  As  a  system  undergoes  design  and  development, 
the  emphasis  in  testing  moves  gradually  from  development  test 
and  evaluation  (DT&E) ,  which  is  chiefly  concerned  with  the 
attainment  of  engineering  design  goals,  to  operational  test 
and  evaluation  (OT&E) ,  which  focuses  on  questions  of 
operational  effectiveness,  suitability,  and  supportability. 

The  integration  of  T&E  requirements  has  several  dimensions 
which  include  two  broad  categories  of  testing:  Government, 
and  contractor.  Government  tests  can  be  further  categorized 
as  user  tests,  which  are  broadly  operational  in  emphasis,  and 
builder  tests,  which  focus  on  achievement  of  development 
requirements. 

Test  and  evaluation  encompasses  the  interrelationships  of 
all  system  elements,  including  equipment,  software, 
facilities,  personnel,  and  procedural  data.  Each  work 
breakdown  structure  (WBS)  element  must  receive  appropriate 
T&E.  In  most  cases  (e.g.,  software)  the  system  element  may 
have  unique  requirements  which  constrain  the  testing  approach. 

Another  T&E  dimension  to  consider  is  that  testing  spans 
the  overall  acquisition  life  cycle.  It  is  not  simply 
something  that  takes  place  when  development  is  complete. 
Finally,  as  T&E  requirements  are  identified  for  the  operation 
and  support  functions,  the  process  can  also  identify  the 
resources  and  procedures  necessary  for  the  test  activities 
themselves. 
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T&E  policy,  described  in  Part  8  of  DOD  Instruction  5000.2, 
provides  guidelines  for  planning  and  conducting  test  and 
evaluation.  It  defines  and  describes  the  major  categories  of 
DT&E  and  OT&E,  and  provides  for  exceptions  such  as  combining 
DT&E  with  OT&E,  T&E  for  special  acquisition  programs,  T&E  of 
computer  software,  T&E  of  system  alterations,  and  joint  T&E 
programs.  DOD  Instruction  5000.2  specifies  three  general 
requirements : 


a.  Successful  accomplishment  of  T&E  objectives  will  be  a 
key  requirement  for  decisions  to  commit  significant 
additional  resources  to  a  program  or  to  advance  it  from  one 
acquisition  phase  to  another. 

b.  T&E  shall  begin  as  early  as  possible  and  be  conducted 
throughout  the  system  acquisition  process  to  assess  and 
reduce  acquisition  risks,  and  to  estimate  the  operational 
effectiveness  and  operational  suitability  of  the  system. 

c.  The  dependence  on  subjective  judgement  of  system 
performance  will  be  minimized  during  testing. 


In  summary,  there  is  clear  policy  stating  test  and 
evaluation  program  requirements,  with  particular  emphasis  on 
those  programs  designated  as  major  weapon  systems.  Test  and 
evaluation  is  an  integral  part  of  the  systems  development 
process.  It  begins  early  and  extends  throughout  the 
acquisition  life  cycle.  The  most  general  objectives  of  the 
T&E  program  are  1)  to  assess  and  reduce  the  risk  to  the 
program  2)  verify  technical  specifications  and  contractual 
guarantees,  and  3)  to  estimate  the  operational  suitability  and 
effectiveness  of  the  system. 
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1.  Devttlopoaental  Test  And  Evaluation 

Developmental  test  and  evaluation  is  conducted 
throughout  the  acquisition  process  to  ensure  the  acquisition 
and  fielding  of  an  effective  and  supportable  system.  DT&E 
includes  test  and  evaluation  of  components  and  subsystems  at 
all  work  breakdown  structure  (WBS)  levels  including  preplanned 
product  improvement  (P3I)  changes,  hardware/software 
integration,  and  related  software  changes,  as  well  as 
qualification,  live  fire,  and  production  acceptance  testing. 
It  involves  the  use  of  simulations,  models,  breadboards, 
brassboards,  and  testbeds,  as  well  as  engineering  and 
manufacturing  development  models  or  prototypes  of  system 
coR^onents  or  the  system  itself. 

DT&E  is  normally  planned,  conducted,  and  monitored  by 
the  developing  agency.  DT&E  is  conducted  to: 

a.  Assist  the  engineering  design  and  development  process. 

b.  Verify  performance  objectives  and  specifications. 

c.  Demonstrate  that  design  risks  have  been  minimized. 

d.  Evaluate  the  compatibility  and  interoperability  with 

existing  or  planned  equipment /systems. 

e.  Provide  an  assurance  that  the  system/equipment  is  ready 

for  testing  in  the  operational  environment. 

DT&E  is  conducted  during  the  Concept  E3q>loration/ 
Definition  (C/E)  phase  to  assist  in  selecting  preferred 
alternative  system  concepts,  technologies,  and  designs.  During 
the  Concept  Demonstration/Validation  (D/V)  phase,  DT&E 
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identifies  and  validates  the  preferred  technical  approach, 
including  the  identification  of  technical  risks  and  feasible 
solutions.  During  the  Engineering  and  Manufacturing 
Development  phase,  DT&E  demonstrates  that  engineering  is 
reasonably  complete,  that  all  significant  design  problems  are 
in  hand,  and  that  the  design  meets  its  required  specifications 
in  all  areas  (such  as  performance,  reliability,  and 
maintainability)  within  the  range  of  environmental  pareuneters 
designed  for  the  operational  employment  of  the  system.  After 
the  Production  and  Deployment  Decision,  DT&E  is  an  integral 
part  of  the  development,  validation,  and  introduction  of 
system  changes  undertaken  to  improve  the  system,  to  react  to 
new  threats,  and/  or  to  reduce  life  cycle  costs. 

2.  Qualification  Testing 

As  part  of  DT&E,  each  developing  agency  is  also 
responsible  for  the  qualification  testing  that  verifies  the 
design  and  the  manufacturing  process  and  provides  a  baseline 
for  subsequent  acceptance  tests.  Qualification  tests  consist 
of  pre-production  and  production  qualification  tests. 

Pre-production  qualification  tests  are  formal 
contractual  tests  that  ensure  design  integrity  over  the 
specified  operational  and  environmental  range.  These  tests 
usually  use  pre-production  or  prototype  hardware  fabricated  to 
the  proposed  production  specifications  and  drawings.  Such 
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tests  include  the  reliability  and  maintainability 
demonstration  tests  required  prior  to  production  release. 

Production  qualification  tests  are  conducted  for  all 
production  items  to  ensure  the  effectiveness  of  the 
manufacturing  process,  equipment,  and  procedures.  All  new 
production  items  are  subjected  to  first  article  tests  to 
verify  specification  compliance  and  form,  fit  and  function. 
Production  acceptance  tests  are  also  conducted  on  each  item  or 
on  a  sample  lot  taken  at  random  from  each  production  lot. 
These  tests  are  repeated  when  the  process  or  design  is  changed 
significantly,  and  when  a  second  or  alternative  source  is 
brought  on  line.  Production  qualification  tests  are  conducted 
against  the  contractual  requirements. 

3 .  Operational  Test  And  Evaluation 

The  primary  purpose  of  operational  test  and  evaluation 
is  to  verify  that  operationally  effective  and  operationally 
suitable  systems  are  approved  for  production  that  meet  mission 
needs  and  minimum  operational  performance  requirements  of  the 
operating  forces. [Ref.  3] 

For  major  systems,  OT&E  is  normally  planned  and 
conducted  by  a  major  OT&E  field  agency  located  within  the  DoD 
component.  This  Operational  Test  Agency  (OTA)  must  be 
separate  and  Independent  from  the  developing/procuring  agency. 
The  OTA  is  responsible  for  managing  operational  testing, 
reporting  test  results,  and  providing  its  independent 
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evaluation  of  the  system  being  tested  directly  to  the  Military 
Service  Chief  or  Defense  Agency  Director. 

The  principal  objectives  of  OT&E  are  to; 

a.  Estimate  the  operational  effectiveness  and  operational 
suitability  of  the  system. 

b.  Identify  needed  modifications  or  improvements. 

c.  Provide  details  on  tactics,  doctrine,  organizational, 
and  personnel  requirements. 

d.  Provide  information  to  uphold  and  verify  the  adequacy 
of  various  manuals,  handbooks,  supporting  plans,  and 
documentation . 

Modeling  and  simulation  can  assist  in  the  T&E  planning 
process  and  can  reduce  the  cost  of  the  conduct  of  testing. 
Areas  of  particular  application  include  scenario  development 
and  the  timing  of  test  events;  the  development  of  objectives, 
essential  elements  of  analysis,  and  measures  of  effectiveness; 
the  identification  of  variables  for  control  and  measurement, 
and  the  development  of  data  collection,  instrumentation  and 
data  analysis  plans.  Modeling  and  simulation  can  be  used  to 
predict  ahead  of  time  the  effects  of  various  assumptions  and 
constraints  and  evaluate  candidate  measures  of  effectiveness 
to  help  in  formulation  of  the  test  design  plan. 

Simulations  are  not  a  substitute  for  live  testing 
since  there  are  many  things  that  cannot  be  adequately 
simulated  by  computer  programs.  Examples  include  the  decision 
process  and  the  proficiency  of  personnel  in  the  performance  of 
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their  functions.  Therefore,  operational  test  and  evaluation 
does  not  include  eui  operational  assessment  based  exclusively 
on  conputer  modeling  or  simulation.  It  also  shouldn't  be 
based  purely  on  an  analysis  of  system  requirements, 
engineering  proposals,  design  specifications,  or  any  other 
information  that  is  contained  in  the  programs 
documents. [Ref.  4] 

Although  OT&E  is  planned  and  conducted  by  an 
independent  testing  activity,  the  Program  Manager  (PM) 
provides  the  funding.  He  must  closely  coordinate  all  aspects 
of  test  and  evaluation  with  this  organization  to  plan 
appropriate  funding  and  to  ensure  that  DT&E  objectives 
coincide  with  OT&E  objectives. 

OT&E  is  conducted  in  an  environment  that  is  as 
operationally  realistic  as  possible.  Typical  operation  and 
support  personnel  are  used  to  obtain  a  valid  estimate  of  the 
user's  capability  to  operate  and  maintain  the  system  when 
deployed  under  both  peacetime  and  wartime  conditions.  During 
operational  testing,  threat  representative  forces  should  be 
used  whenever  possible.  The  items  tested  must  sufficiently 
represent  expected  production  models  to  ensure  that  a  valid 
assessment  of  the  system  can  be  made. 

Normally,  limited  follow-on  operational  testing 
(POT&E)  will  use  the  same  system  and  support  equipment  used  in 
the  operational  evaluation  and  will  test  the  fixes  to  be 
incorporated  in  production  systems,  complete  deferred  or 
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incomplete  pre-production  test  and  evaluation,  and  continue 
tactics  development.  FOT&E  will  continue  until  the  objectives 
specified  in  the  approved  TEMP  for  this  phase  have  been  met, 
regardless  of  the  date  of  deployment  of  the  production 
systems . 

Other  operational  testing  may  include  tests  of  the 
existing  system  in  a  new  environment,  with  a  new  subsystem,  in 
a  new  tactical  application,  or  against  a  new  threat.  This 
also  includes  system  upgrades  as  well  as  changes  made  to 
correct  deficiencies  identified  during  previous  test  and 
evaluation.  [Ref.  5]  Excunples  of  this  include  all 
hardware  and  software  alterations  that  materially  change 
system  performance  (operational  effectiveness  and  suitability) 
and  must  therefore  be  adequately  tested  and  evaluated. 

4 .  Combined  Testing 

Since  DT&E  and  OT&E  take  place  during  the  same  phases 
of  the  acquisition  cycle,  it  may  make  sense  to  coordinate 
developmental  and  operational  testing  to  use  resources  more 
efficiently  in  obtaining  the  data  necessary  to  satisfy  the 
common  needs  of  both  the  developing  agency  and  the  operational 
test  agency.  This  is  called  combined  testing.  Development 
euid  operational  tests  can  be  combined  with  approval,  when 
significant,  clearly  identified  cost  and  time  benefits  will 
result.  Of  course,  the  test  objectives  of  both  the  developing 
agency  and  the  operational  test  agency  will  have  to  be 
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reflected  in  this  combined  testing  situation.  At  the 
conclusion  of  testing,  separate  DT  and  OT  reports  must  be 
submitted. 

D.  THE  TEST  AND  EVALUATION  MASTER  PLAN 

A  major  controlling  document  for  every  acquisition  progreun 
is  the  Test  and  Evaluation  Master  Plan  (TEMP) ,  which  lays  out 
an  overall  plan  for  developmental  and  operational  test  and 
evaluation,  designed  to  verify  that  the  new  equipment  meets 
the  requirements. 

The  TEMP  documents  the  overall  structure  and  objectives  of 
the  test  and  evaluation  program  [Ref .  6]  .  It  provides 
a  frcunework  within  which  to  generate  detailed  test  and 
evaluation  plans  and  it  documents  schedule  and  resource 
inplications  associated  with  the  T&E  program.  The  TEMP 
identifies  the  necessary  DT&E  and  OT&E  activities.  It  relates 
program  schedule,  test  management  strategy  and  structure,  and 
required  resources  to; 

(1)  Critical  operational  issues; 

(2)  Critical  technical  pareuneters ; 

(3)  Minimum  acceptable  performance  requirements; 

(4)  Evaluation  criteria;  and 

(5)  Milestone  decision  points. 
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The  TEMP'S  five  part  format  is  detailed  in  DoD  5000. 2 -M, 
as  follows: 


Part  I  •  Systaa  Outllna 

•  Mission  Description 

e  System  Threat  Assessment 

e  Minimum  Acceptable  Operational  Performance  Requirements 
e  System  Description 
e  Critical  Technical  Parameters 

Part  II  -  Integrated  Test  Program  Summary 
e  Integrated  Test  Program  Schedule 
e  Mauiagement 

Part  III  •  DTRB  Outllna 

e  DT&E  Overview 
e  DT&E  to  Date 
e  Future  DT&E 

e  Live-Fire  Test  &  Evaluation 

Part  rv  -  0T6B  Outline 

e  OT&E  Overview 
e  Critical  Operational  Issues 
e  OT&E  to  Date 

•  Future  OT&E 

Part  V  •  Test  and  Evaluation  Resources 
e  Test  Articles 

•  Test  Sites  and  Instrumentation 
e  Test  Support  Equipment 

e  Threat  Systems /Simulators 
e  Test  Targets  and  E}q>endables 
e  Operational  Force  Test  Support 
e  Simulators,  Models  and  Testbeds 
e  Special  Requirements 
e  Manpower/Personnel  Training 

Figure  1 

Part  I  concerns  system  details  including  production  and 
delivery  Information  and  the  operational  and  technical  goals 
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euid  thresholds.  Part  II,  program  summary,  includes  a  schedule 
chart  that  provides  an  overview  of  the  major  acquisition  and 
T&E  events.  Parts  III  (DT&E  Outline)  and  IV  (OT&E  outline) 
describe  in  cjuantitative  terms  the  scope  of  each  major  test 
period.  Part  V,  the  Test  Resource  Summary,  identifies  special 
resources  required  for  the  test  program. 

The  TEMP  is  a  dynamic  document  that  should  be  updated  at 
milestones  and  whenever  the  program  has  changed  significantly. 
Its  contents  should  be  factual  and  specific,  avoiding 
generalities,  and  eit^hasizing  quantifiable  and  testcUsle 
requirements,  both  operational  and  technical.  Although  a 
summary  document,  it  is  imperative  that  pertinent,  but 
integrated,  facts  and  descriptions  be  included.  The  contents 
must  describe  the  amount  and  type  of  system  testing  to  be 
conducted  before  each  milestone,  and  the  resources 
required. [Ref.  7] 

B.  TEST  RESOURCES 

It  is  important  that  the  necessary  test  resources  be 
identified  early  in  the  acquisition  process.  These  resources 
include : 

(1)  Test  Articles.  The  PM  should  identify  the  actual 
number  and  timing  requirements  for  all  test  articles, 
including  key  support  equipment.  Specifically  identify 
when  prototype,  engineering  development,  preproduction,  or 
production  models  will  be  used. 
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(2)  Z££i£ _ _ and  Instrumentation.  Identify  the 

specific  test  ranges/facilities  to  be  used  for  each  type  of 
testing.  Compare  the  test  requirements  against  the 
range/facility  capabilities  and  identify  any  major 
shortfalls.  Also,  identify  instrumentation  that  must  be 
acquired  specifically  to  conduct  the  planned  tests. 

(3)  Test  Support  Equipment .  Identify  test  support 
equipment  that  must  be  acquired  specifically  to  conduct  the 
test  program. 

(4)  Ibr.ga.t _ Systems /Simulators .  Identify  the  type, 

number,  availcdjility,  and  fidelity  requirements  for  all 
threat  systems/simulators.  Compare  the  requirements  for 
all  threat  systems  with  available  and  projected  assets  and 
their  capabilities.  Highlight  any  major  shortfalls.  Each 
threat  simulator  shall  be  subjected  to  validation 
procedures  to  estcdslish  and  document  a  baseline  comparison 
with  its  associated  threat  and  to  ascertain  the  extent  of 
the  operational  and  technical  performance  differences 
between  the  two  throughout  the  simulator's  life- cycle. 

(5)  Test  Targets  and  Expendables.  Identify  the  type, 
number,  and  availability  requirements  for  all  targets, 
flares,  chaff,  smoke  generators,  kill  indicators,  etc.  that 
will  be  required  for  each  phase  of  testing.  Identify  any 
major  shortfalls. 

(6)  Operational  Force  Test  Support.  For  each  test  and 
evaluation  phase,  identify  the  type  and  timing  of  aircraft 
flying  hours,  communications  station  support,  on-orbit 
satellite  contacts/coverage,  and  other  critical  support 
required . 

(7)  Simulations.  Models  and  Testbeds.  For  each  test  and 
evaluation  phase,  identify  the  system  simulations  required, 
including  computer -driven  simulation  models  and 
hardware/ software -in -the -loop  testbeds.  Identify  the 
resources  required  to  validate  and  certify  their  credible 
usage  or  application  before  their  use. 

(8)  Special  Requirements.  Discuss  requirements  for  any 
significant  non- instrumentation  capabilities  and  resources 
such  as:  special  data  processing/data  bases,  unique 
mapping/charting/geodesy  products,  extreme  physical 
environmental  conditions  or  restricted/special  use 
air/sea/landscapes . 

(9)  Test  and  Evaluation  Funding  Requirements.  Estimate, 
by  Fiscal  Year  and  appropriation  line  number  (progreun 
element) ,  the  funding  required  to  pay  direct  costs  of 
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planned  testing.  State,  by  fiscal  year,  the  funding 
currently  appearing  in  those  lines  (program  elements) . 
Identify  any  shortfalls. 

(10)  Manpower /Personnel  Training.  Identify  manpower/ 
personnel  and  training  requirements  and  limitations  that 
affect  test  and  evaluation  execution. 

As  system  development  progresses,  the  preliminary  test 
resource  requirements  should  be  reassessed  and  refined  and 
TEMP  updated  to  reflect  any  changed  system  concepts.  Resource 
shortfalls  which  Introduce  significant  test  limitations  should 
also  be  discussed,  along  with  an  outline  of  planned  corrective 
action. [Ref.  8] 

The  PM  is  responsible  for  developing  the  TEMP,  including 
its  content  and  preparation.  However,  since  part  IV  concerns 
OT&E,  the  Operational  Test  Agency  (OTA)  is  usually  responsible 
for  the  preparation,  content,  and  coordination  of  that  part  of 
the  TEMP.  Therefore,  the  PM  must  establish  early  liaison  with 
the  operational  tester.  This  assists  the  OTA  with  complete 
integration  of  operational  assessments  and  OT&E  requirements 
into  the  TEMP. [Ref .  9] 

F.  PER80MMEL  AMD  TRAZMIM6 

It  is  DoD  policy  that  manpower,  personnel,  and  training 
(MPT) ,  as  essential  elements  of  integrated  logistics  support 
(ILS) ,  be  given  explicit  attention  early  in  the  acquisition 
process.  Principal  activities  required  include  determination 
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and  specification  of  requirements  based  on:  previous 
experience  with  similar  systems,  demographic  expectations, 
design  trade-offs,  and  contractor  incentives  to  meet  MPT 
objectives. 


1 .  Personnel 

Prior  to  OT&E,  several  categories  of  personnel  must  be 
identified,  trained  and  available  when  the  system  is  ready  to 
test.  Billet  requirements  must  be  identified  and  funded  for 
the  following  personnel  categories: 


•  Installation  technicians  to  design  and  maintain  test 
support  equipment  and  instrumentation. 

•  Operations  personnel  to  participate  in  and  support  the 
test  as  'enemy'  forces,  and  as  typical  users  manning  the 
tested  system. 

•  Maintenance  technicians  to  maintain  the  tested  equipment 
(users) ,  and  to  maintain  mockups  and  simulators  that  are 
used  in  support  of  the  test. 

•  Supervisors  to  direct,  oversee,  support  and  validate  the 
tests. 

•  Transportation  personnel  to  test  the  transportability  of 
the  system  by  land,  sea  and  air. 


Each  of  these  categories  of  personnel  may  require: 


•  Training  personnel/instructors  to  conduct  support  and  user 
training  prior  to  OT&E. 

•  Training  programs  of  instruction  which  must  be  developed 
for  instructor  use;  and 
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•  Security  clearances  to  allow  necessary  personnel  access  to 
sensitive  or  classified  data  which  are  used  in  the  system 
itself,  or  gathered  as  a  result  of  the  testing. 

The  PM  must  identify  the  personnel  requirements  and 
skill  levels  necessary  for  the  system/equipment  under  all 
normal  conditions  of  readiness.  Operational  manning  and 
maintenance  manpower  requirements  at  all  applicable 
maintenance  levels  must  be  addressed.  Manning  levels  and 
schedules  should  be  Identified  by  maintenance  level  for  each 
euiticipated  field  testing  site.  It  is  necessary  to  identify 
all  training  courses  required  for  installation,  operation  and 
maintenance  personnel,  together  with  locations  and  duration  of 
each  course.  Special  training  devices  required  for  such 
courses  at  each  location  should  be  identified  and  procured. 

2 .  Training 

Training  and  training  support  includes  the  processes, 
procedures,  techniques,  training  devices  and  equipment  used  to 
train  civilian,  active  duty  and  reserve  military  personnel  to 
operate  and  support  a  material  system.  This  includes: 
individual  euid  crew  training;  new  equipment  training;  initial, 
formal,  and  on-the-job  training;  and  logistic  support  planning 
for  training  equipment  and  training  device  acquisitions  and 
installations.  [Ref.  10] 
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O.  TBCHNZCAL  MANUALS 


Technical  manuals  (TMs)  and  publications  are  an  integral 
part  of  the  system  /equipment  support  requirements.  They  are 
the  prime  means  of  communicating  maintenance  and  operation 
information  to  the  user.  Manual  requirements  must  be  planned, 
progressively  monitored  and  updated  to  ensure  timely 
completion  and  delivery  for  adequate  logistics  support.  Since 
the  quality  of  TMs  affects  equipment  maintainability, 
personnel  efficiency,  safety  and  readiness,  quality  in  TMs 
must  be  a  planned  objective.  [Ref.  11] 

Prior  to  entering  into  a  formal  arrangement  for  a 
contractor  to  produce  TMs,  the  Government  is  responsible  for 
furnishing  guidance  to  the  contractor  for  the  development  of 
a  Technical  Manual  Plan  (TMP) .  The  contractor  has  the 
responsibility  of  justifying  and  validating  each  manual 
recommended.  Additional  contractor  responsibilities  include 
providing  engineering  design,  maintainability,  and  maintenance 
analysis  documentation  and  assistance. 

The  contractor  must  develop  and  implement  a  validation 
plan  that  the  Government  formally  approves.  The  validation  of 
technical  manuals  is  a  continuing  effort  that  the  Government 
is  required  to  verify.  This  verification  refers  to  the 
adequacy  and  accuracy  of  the  manuals. 
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ZZZ.  FORT  HnNTER-LZGGETT  OBSERVATZOM8 

A.  FORT  HUMTER-LZGOET  EZFERZMEMTATZOH  CENTER 

The  TEXCOM  Experimentation  Center  (TEC)  was  established  at 
Fort  Hunter-Liggett  in  1956.  Since  that  time,  it  has  existed 
in  various  sizes  and  configurations,  but  always  with  the  same 
general  mission  of  testing  new  Army  weapon  systems.  The 
TEXCOM  Experimentation  Center's  specific  mission  is  to: 

a.  Plan,  conduct,  and  report  on  Army  user  tests  and 
experiments  of  doctrine,  training,  organization,  and  material. 

b.  Provide  advice,  assistance,  and  guidance  on  test  and 
experimentation  to  combat  developers,  training  developers, 
material  developers,  system  managers,  material  producers, 
other  military  services,  and  private  industry. 

c.  Conduct  other  tests  and  experiments  as  directed  by 
Commander,  TEXCOM. 

d.  Provide  high  resolution  data,  other  scientifically 
derived  data,  and  analysis  of  these  data  for  training, 
doctrine,  organization,  and  material  development  decisions. 

e.  Maintain  a  highly  responsive,  trained,  combat  ready 
reinforced  tank  company  (M>1  Abrams  and  M-2  Bradley)  for 
experimentation  and  testing  support. 

f.  Within  TEXCOM,  develop  instriunentation  programs,  test 
and  experimentation  methodologies  and  design,  develop. 
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procure,  and  maintain  the  instrumentation  necessary  to 
generate  required  data  from  field  experiments. 

The  TEC  facility  is  coit^iosed  of  instrumented  range 
facilities  that  permit  extensive  and  detailed  computer  data 
collection  in  support  of  its  mission.  TEC  is  a  self  reliant 
organization  that  is  fully  capable  of  supporting  most  testing 
requirements.  This  includes  the  on  site  design  and 
manufacturing  of  training  aids  and  devices  which  are  necessary 
for  successful  testing.  To  facilitate  future  operational 
tests,  TEC  is  developing  a  portable  range  instrumentation 
system  that  will  permit  testing  to  be  conducted  at  locations 
other  than  Fort  Hunter- Ligget. 


B.  THE  APACHE  HELICOPTER 
1 .  Background 

The  earliest  system  test  examined  by  this  thesis  is 
the  Apache  Advanced  Attack  Helicopter  (AAH)  OT  II,  which 
occurred  from  June  to  August  1981.  Operational  testing  for 
the  Apache,  designated  the  AH- 64  Attack  Helicopter,  was  a 
Follow-on  Operational  Test  &  Evaluation  (FOT&E) .  The  initial 
operational  test  (lOT/OT  I)  was  conducted  in  1976.  It  had 
compared  two  candidate  systems  with  their  respective 
baselines,  each  an  AH- IS  Cobra,  and  led  to  selection  of  the 
AH- 64  for  further  development. 
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The  Apache  OT  II  was  a  comparative  baseline,  three- 
phase  test  conducted  at  the  Test  and  Evaluation  Center  (TEC) . 
A  typical  Army  attack  helicopter  unit  provided  personnel  and 
resources  for  both  an  AH-64  test  section  and  the  AH-IS 
baseline  section.  The  test  section  consisted  of  three  AH-64s 
and  two  Airborne  Target  and  Fire  Control  Systems  equipped  AH- 
ISs  to  act  as  scouts  for  the  AH-64s. 

The  baseline  section  consisted  of  three  AH-lSs  and  two 
OH-58  scouts.  The  AH-IS  and  AH-64  aircraft  were  flown  in  the 
same  operational  and  threat  environment. 

The  three  phases  of  the  test  included  a  training 
phase,  a  non-live  fire  phase,  and  a  live  fire  phase.  Force- 
on-force  and  one-on-many  engagements,  with  real  time  casualty 
assessment,  were  conducted  during  the  non-live  fire  phase. 
The  live  fire  phase  included  firing  of  all  AAH  weapons.  In 
total,  over  400  flight  hours  were  accomplished. 

The  purpose  of  the  test  was  to  assess  the  military 
effectiveness  of  the  AH-64  against  the  baseline  aircraft  it 
was  to  replace.  The  OT-II  operational  test  report  stated: 
"the  performance  of  the  AH-64  was  adequate  for  combat, 
superior  to  the  present  attack  helicopters,  night  capable,  and 
survivable. '*  There  were  no  operational  issues  which  were 
considered  to  preclude  the  acquisition  and  development  of  the 
system. 

DT  II  was  conducted  after  OT  II  in  June,  1982.  The 
purpose  was  to  fix  performance  problems  and  operational 
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failures  which  had  occurred  during  the  operational  test.  This 
primarily  involved  the  Target  Acquisition  Designation  Sight 
(TADS)  which  had  failed  to  perform,  thus  affecting  testing  of 
the  aircraft  weapon  systems. 

2 .  Observations 

Since  there  were  many  new,  highly  sophisticated 
systems  from  eight  different  PM  offices  integrated  on  the  AH- 
64,  a  separate  development  test  training  detachment  was  formed 
to  prepare  for  OT  II.  Preliminary  flight  training  was 
performed  using  modified  Cobra  (AH-1)  surrogate  aircraft. 
Eventually,  fifteen  prototype  training  devices  were  developed. 
Thirteen  were  for  the  support  of  maintenance  training,  and  two 
were  for  support  of  pilot/gunner  training. 

To  support  OT  II,  the  Apache  PM  established  a  field 
office  at  the  test  area.  Included  in  this  office  were  PMO 
personnel  from  the  logistics,  test  and  evaluation,  and 
technical  divisions.  Although  controlled  by  test  personnel, 
these  PM  representatives  were  able  to  improve  test  continuity 
and  facilitate  the  flow  of  spares  and  repair  parts. 
Controlling  spares  and  parts  had  the  additional  benefit  of 
helping  to  keep  PMO  personnel  informed  of  what  was  going  on. 

Some  contractor  personnel  were  permitted  at  the  OT  II 
test  site  to  represent  the  depot  level  of  maintenance  support. 
These  personnel  included  technical  writers  whose  job  had  been 
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to  write  the  AH-64  Technical  Manuals  (TMs) .  During  the 
operational  tests,  the  technical  writers  were  able  to  evaluate 
recommended  TM  changes  from  crew  members  and  support 
personnel,  and  update  publications  on  the  spot.  These  updates 
were  then  passed  back  to  the  users  during  the  test. 

During  the  operational  test,  the  Target  Acquisition 
Designation  System  (TADS)  was  undergoing  developmental  and 
operational  testing  at  the  same  time.  The  development 
schedule  hadn't  allowed  enough  time  for  qualification  testing 
(a  DT&E  activity)  of  the  TADS  prototype  prior  to  a  full  field 
test  of  the  total  aircraft  system.  There  also  hadn't  been 
time  to  introduce  changes  to  correct  TADS  problems  discovered 
in  early  developmental  tests.  As  a  result,  the  TADS  performed 
poorly  and  was  unreliable  during  the  operational  test. 

Problems  with  the  weapon  systems  were  experienced  from 
the  start  of  the  weapons  subtests,  and  only  one  trial  was 
successfully  completed  with  no  aborts  or  difficulties. 
Weapons  control  failures,  aircraft  generator  failures,  random 
pylon  articulation,  and  dirt  inside  the  30mm  gun's  receiver 
group  were  all  problems  that  curtailed  testing. 

These  failures  Included  the  2.75"  Folding  Fin  Aerial 
Rocket  (FFAR)  engagements  which  experienced  widely  dispersed 
impacts  over  a  2  kilometer  area.  This  was  attributed  to 
problems  with  the  TADS  and  articulation  of  the  weapons 
pylon. [Ref.  12]  Also,  due  to  the  limited  firing  of 
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the  SOmrn  gun  and  the  FFAR,  only  one  simulated  engagement  with 
the  new  HELLFIRE  missile  was  accon^lished. 

The  i^ache  tests  also  experienced  significant  test 
limitations  due  to  ineffective  threat  weapons  simulators.  The 
SA-9  simulator,  a  air  defense  missile  system,  could  not  be 
employed  quickly,  and  it  constantly  overheated.  The  ZSU-23-4 
simulator,  a  threat  air  defense  gun,  wasn't  capsdale  of  firing 
on  the  move.  The  T-72  tank  surrogates  couldn't  fire  on  the 
move  either.  For  these  reasons,  they  did  not  represent  a 
realistic  threat  force. 

Operational  testing  for  the  Apache  concluded  in  August 
1981.  Results  of  the  operational  test  showed  that  the  system 
was  superior  to  the  Cobra  attack  helicopter,  night  capable, 
and  survivable.  At  Milestone  III,  the  Apache  was  approved  for 
production. 

C.  APACHE  ENHANCEMENT  OBSERVATIONS  AND  LESSONS  LEARNED 
1.  Test  Schedules 

Lesson  Learned.  Test  schedules  should  be  delayed  or  modified 
if  the  system  isn't  ready  for  the  operational  test. 
Discussion.  The  TADS  on  the  Apache  hadn't  completed  DT&E,  and 
it  wasn't  ready  for  OT&E.  This  had  a  negative  affect  on  the 
results  of  all  three  weapons  sxibsystem  tests.  The  30mm  gun 
failures  were  partially  attributed  to  the  TADS  failure,  but 
that  blame  may  have' been  misplaced.  As  recently  as  Operation 
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Desert  Storm,  the  Apache  continued  to  experience  trouble  with 
the  30mm  gun.  [Ref .  13]  This  problem  might  not  have 
occurred  if  the  PM  had  delayed  testing  until  the  complete 
system  was  ready. 

2 .  Teehnioal  Manuals 

Enhancement  Observation.  Contractor  technical  writers  were 
included  at  the  operational  test  site. 

Discussion.  Having  the  technical  writers  present  for  OT&E 
ensured  that  user  input  was  taken  seriously.  Problems  with 
the  TMs  were  fixed  and  the  PM  improved  the  quality  of  his 
system  at  the  test  site. 

3 .  Test  Reports 

Lesson  Learned.  Test  reports  should  include  user  comments  and 
a  sufficient  discussion  of  problems  found  during  the  test. 
Discussion.  The  Apache  test  report  lacked  detail  in  both  of 
these  areas.  These  comments  are  very  important  in  enabling 
non-participants  to  gain  an  understanding  of  the  performance 
of  the  weapons  system. 

4 .  Test  Resources 
a.  Tost  Articles 

Lesson  Learned.  Operational  testing  should  not  be  conducted 
on  multiple  new  systems  at  the  same  time. 
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Discussion.  The  Apache  was  composed  of  multiple  new  systems 
integrated  together.  The  failure  of  any  of  these  systems 
affected  the  test  data  collection  on  the  others.  In  this 
test,  the  TADS  was  not  prepared  for  the  test,  and  it  had  a 
negative  affect  upon  results  from  the  30mm  gun,  the  rocket 
system,  and  the  Hellfire  missile. 

Jb.  Threat  Systeoa/ Simulator  a 

Lesson  Learned.  Threat  systems  should  adequately  represent 
the  threat  forces. 

Discussion.  Some  limitations  were  reported  with  the  threat 
systems  due  to  their  inaJDility  to  react  quickly  and  fire  on 
the  move.  This  gave  the  Apache  helicopters  less  incentive  to 
mask  or  fire  quickly  as  they  would  in  a  realistic  environment. 
This  could  have  resulted  in  invalidated  test  engagements,  and 
overstated  the  combat  effectiveness  of  the  system 

5.  User  E3q>«rlence  Training 

Enhancement  Observation.  Extensive  user  training  was 
en^hasized  prior  to  the  operational  test. 

Discussion.  The  Apache  crews  were  formed  into  a  separate  test 
training  detachment  to  prepare  for  OT&E  and  preliminary  flight 
training  «ras  conducted  on  modified  Cobra  helicopters.  To 
support  this  training,  the  Apache  progreun  developed  a  total  of 
fifteen  prototype  training  devices.  Thirteen  of  these  were 
dedicated  to  the  support  of  maintenance  training.  The 
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operational  test  report  did  not  cite  any  OT&E  problems  that 
were  due  to  a  lack  of  prior  training. 

D.  THE  OH-58D  HELICOPTER 

1.  Background 

The  Kiowa  helicopter  is  a  scout  aircraft  that  has  been 
in  the  Army  inventory  for  several  years.  The  decision  to 
modify  the  aircraft  from  the  OH-58C  to  the  OH-58D  in  the 
Advanced  Helicopter  Improvement  Program  (AHIP)  resulted  in  a 
major  weapon  system  upgrade.  This  required  Follow-on 
Operational  Testing. 

Operational  Testing  (OT  II)  for  the  OH-58D  was 
conducted  at  Fort  Hunter-Liggett  from  September  1984  to 
February  1985.  The  test  at  Fort  Hunter-Liggett  immediately 
followed  Developmental  Testing  which  had  been  conducted  Yuma 
Proving  Ground  in  Arizona. 

2 .  Observations 

Although  the  Operational  Test  of  the  0H-58D  was 
considered  to  be  successful,  a  number  of  issues  were  noted  as 
having  adversely  affected  its  conduct. [Ref .  14]  The 
first  issues  involved  the  pilot  and  support  personnel 
individual  skills  Instruction  which  was  received  at  pre¬ 
factory  training,  factory  training,  and  Developmental  Testing. 
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The  Instxxiction  was  judged  to  be  inadequate  in  the  following 
four  areas: 


a.  Not  all  recjuired  individual  tasks  were  taught 

b.  Combat  skills  were  not  taught 

c.  Academics  during  pre- factory  training  were  not 
reinforced  with  either  flight  time  or  training  devices 

d.  Not  enough  flight  or  hot  cockpit  time  was  available  for 
factory  training  due  to  the  fixed  price  contract. 

Many  other  factory  training  problems  occurred.  These 
included  the  lack  of  procedural  trainers.  The  factory  had  one 
actual  OH-58D  availcdsle,  and  this  was  used  for  all  pilot  and 
support  personnel  training.  The  repeated  system  start-ups 
resulted  in  frequent  system  failures  and  caused  training 
delays.  All  other  training  was  conducted  on  chalk  boards. 

Another  issue  was  that  this  factory  training  was 
conducted  from  two  to  five  months  before  the  test.  After 
training  was  completed  the  students  were  sent  back  to  their 
home  units  where  they  resumed  their  normal  duties.  By  the 
time  of  the  test,  many  had  forgotten  much  of  what  they  had 
learned . 

A  positive  aspect  of  the  factory  training  involved 
user  input  to  improve  the  technical  manuals.  The  contractor. 
Bell  Helicopter,  had  technical  writers  present  at  the 
training.  They  were  eUcle  to  incorporate  changes  and  fix 
problems  on  the  spot,  as  they  were  noted  by  mechanics  and 
pilots. 
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other  problems  that  affected  testing  included  the 
availability  of  only  two  training  areas  for  the  first  two 
weeks  of  crew  training  prior  to  the  test.  This  was 
accompanied  by  a  prohibition  from  using  the  system's  lasers 
for  the  same  time  period. 

Technical  restrictions  also  affected  the  test 
schedule.  One  of  these  involved  a  requirement  for  the 
issuance  of  Airworthiness  Releases  for  OH-58C/D  pink  lights 
(Night  vision  Goggle  search  lights) ,  which  delayed  some 
training.  Also,  the  installation  and  verification  testing  of 
test  instrumentation  equipment  on  the  aircraft  caused  further 
delays.  Finally,  one  day  prior  to  the  start  of  the  training 
program,  the  parent  unit  of  the  test  personnel  decided  to 
establish  the  following  additional  requirements  for  single 
pilot  NVG  flight  in  the  OH-58C:  (1)  radar  altimeters,  (2) 
blue-green  level  lighting,  and  (3)  selected  experienced  crews. 
This  created  additional  delays  in  training  until  the 
appropriate  requirements  could  be  met. 

Another  problem  area  involved  crew  qualification  to 
participate  in  the  test.  The  operational  test  plan  called  for 
OH-58C  crews  who  had  gone  through  a  Army  Training  Evaluation 
Program  (ARTEP)  with  the  AH-IS  Cobra  attack  helicopter  before 
the  test.  The  crews  weren't  qualified  as  required  when  they 
arrived  at  the  test  site.  This  made  additional  training 
necessary. 
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In  October  1984,  approximately  one  month  into  testing, 
Headquarters,  Department  of  the  Army  tasked  Fort  Hunter- 
Liggett  with  an  additional  system  evaluation.  This  involved 
a  Scout/Gun  mix  sensitivity  experiment.  The  Test  and 
Evaluation  Center  delayed  testing  for  several  days  to  develop 
instrumentation  for  this  unplanned  test.  They  subsequently 
conducted  18  trials  of  which  only  12  were  technically 
validated.  Tactically,  none  of  these  additional  trials  were 
validated,  so  the  test  results  were  of  limited  use. 

These  test  delays  and  additional  requirements  had  an 
affect  upon  the  test  schedule.  The  trials  were  originally 
expected  to  be  conducted  two  times  a  day  for  four  days  a  week. 
This  was  accelerated  to  two  to  three  trials  a  day,  seven  days 
a  week.  This  impacted  on  staffing  support,  which  was  too 
small  for  the  busier  schedule.  The  aviation  support  unit  was 
kept  extremely  busy  by  this  unexpected  schedule  and  suffered 
from  morale  problems. 

Spare  parts  supply  also  had  some  difficulties  due  to 
post  DT&E  maintenance  requirements.  DT  II  had  concluded  at 
Y\juna,  Arizona  in  August,  and  the  maintenance  that  resulted  had 
drawn  heavily  upon  the  spare  parts  available  for  the 
operational  test.  The  unexpected  OT  II  training  and  testing 
schedule  further  degraded  the  supply  of  spare  parts .  The 
exact  affect  of  this  shortage  was  not  evaluated,  but  comments 
in  the  test  report  stated  that  it  created  a  heavier  workload 
for  the  support  personnel. 
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Weapons  simulator  problems  also  occurred.  TEC 
personnel  believed  that  the  Air  Defense  Threat  Simulators, 
which  were  a  contractor  responsibility,  weren't  up  to  the  test 
requirements.  The  simulators  were  supposed  to  replicate 
current  threat  air  defense  systems  that  would  simulate  a 
realistic  operational  environment  for  flight  crews.  The 
systems  had  technical  problems  as  well  as  crew  training  and 
operational  difficulties.  This  resulted  in  few  valid 
engagements  against  the  0H-58DS  being  tested  and  evaluated. 

Operational  testing  for  the  OH-58D  concluded  in 
February  1985.  At  the  Milestone  III  decision,  the  system  was 
approved  for  fielding. 

E.  OH-58D  EMHAMCEMEMT  OBSERVATIONS  AMD  LESSONS  LEARNED 
1.  Test  Schedules 

Lesson  Learned .  OT&E  should  not  be  scheduled  too  closely 

behind  a  developmental  testing  event. 

Discussion.  The  OH-58D  went  into  its  operational  test  one 
month  behind  its  developmental  test  at  Yuma.  The  quick 
transition  didn't  allow  time  to  correct  system  problems  or  to 
fully  recoup  the  spare  parts  supply  from  the  post-DT&E 
maintenance.  This  caused  extra  work  for  the  OT&E  support 
personnel,  and  it  could  have  negatively  affected  the 
Reliability,  Availability  and  Maintainability  (RAM) 
data  collected  during  the  test. 
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Lesson  Learned.  Test  requirements  should  not  be  added  once 
the  schedule  has  been  established,  and  especially  after 
testing  has  started. 

Discussion.  The  last  minute  requirement  for  a  Scout /Gun 
sensitivity  test  severely  taxed  the  resources  of  Fort  Hunter- 
Liggett,  and  resulted  in  worn-out  crews  and  support  personnel. 
This  extra  requirement  resulted  in  the  collection  of  data  that 
were  not  validated  and  thus,  were  of  little  use.  The  schedule 
deviation  did  have  a  negative  affect  upon  the  test  through  the 
long  hours  and  overwork  that  resulted. 

Lesson  Learned.  User  training  time  should  be  planned  into  the 
beginning  of  the  test  schedule. 

Discussion.  with  this  system,  prior  user  training  was 
insufficient.  This  resulted  in  a  need  for  significant  and 
unexpected  retraining  at  Fort  Hunter-Ligget  prior  to  the  test. 
This  time  was  taken  from  the  test  schedule,  and  contributed  to 
the  overwork  of  the  support  personnel. 

2 .  Technical  Manuals 

Enhancement  Observation.  Early  efforts  were  taken  to  review 
and  correct  the  TMs. 

Discussion.  The  contractor  took  advantage  of  the  factory 
training  to  have  the  technical  writers  and  users  find  and  fix 
TM  problems  before  the  operational  test.  These  problems  could 
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have  had  an  adverse  affect  upon  the  OT&E  RAM  data  if  they 
hadn't  been  corrected  prior  to  the  tests. 

3 .  Test  Reports 

Enhancement  Observation.  A  detailed  test  report  can  be  of 
great  value  in  determining  problems  and  for  developing  a 
programs  ' lessons  learned ' . 

Discussion.  This  thesis  found  the  OH-58D  test  report  to  be 
the  most  thorough  of  the  four  systems  examined.  It  had 
detailed  user  comments,  and  it  traced  test  problems  back  to 
their  sources.  This  resulted  in  a  extremely  informative  test 
report . 

4 .  Test  Resources 

a.  Test  Sites  and  Instrumentation 
Lesson  Learned .  Sufficient  training  areas  should  be  made 

available  at  the  operational  test  site. 

Discussion.  Pre-training  at  Fort  Hunter-Ligget  was  initially 
hindered  by  limited  training  areas  and  the  inability  to 
utilize  the  laser  system  on  the  aircraft.  For  the  OH-58D, 
pre-training  was  critical  to  the  success  of  the  weapons 
system. 


b.  Test  Support  Equipment 

Lesson  Learned.  Spare  parts  supplies  should  be  available  to 
support  testing  operations. 
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Discussion.  The  OH-58D  spare  parts  supply  was  depleted  by 
post-DT&E  maintenance.  This  could  have  impacted  on  the  OT&E 
RAM  data  and  unnecessarily  Increased  program  risk. 

c.  Threat  Systsms/ Simulators 

Lesson  Learned.  Threat  simulators  should  be  adequate  for  the 
tests . 

Discussion.  The  threat  systems  in  this  test  did  not  simulate 
an  effective  threat  environment.  This  was  due  to  mechanical 
and  crew  training  problems  on  the  threat  systems.  These 
problems  could  have  invalidated  the  tests. 

d.  Operational  Force  Test  Support 

Lesson  Learned.  Coordinate  with  test  support  units  early  on, 
so  that  they  can  voice  any  concerns. 

Discussion.  The  test  support  unit  had  legitimate  safety 
concerns  over  night  flying  of  the  weapon  system,  but  the  time 
to  discover  a  problem  is  not  the  day  before  the  test.  If  the 
support  unit  has  a  full  understanding  of  what  is  going  to 
occur,  they  can  give  their  input  and  make  objections  early  in 
the  process. 

Lesson  Learned.  Sufficient  support  personnel  to  meet  test 
requirements  should  be  available. 

Discussion.  There  were  insufficient  support  personnel 
involved  in  this  test  to  handle  the  work  load  brought  on  by 
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the  additional  training.  This  shortfall  could  have  negatively 
affected  maintainability  data  and  should  be  avoided  in  future 
test  programs. 

5.  User  Baq>erlence  Training 

Lesson  Learned.  Users  should  have  sufficient  technical  and 
tactical  training  before  the  tests  are  scheduled  to  begin. 
Discussion.  Even  though  this  system  was  an  upgrade  of  the  OH- 
58C,  significant  technical  and  tactical  differences  affected 
the  users.  They  were  not  qualified  as  required,  and  the 
factory  training  was  insufficient.  This  resulted  in  extra  on¬ 
site  training. 

Lesson  Learned .  Testing  should  follow  soon  after  user 

training  is  conducted. 

Discussion.  With  this  system,  a  three  to  five  month  lag 
occurred  between  training  and  the  tests.  The  need  to  conduct 
retraining  used  up  several  days  of  the  available  test  time. 

Lesson  Learned.  Sufficient  simulators  should  be  available  for 
crew  and  support  training  prior  to  the  test. 

Discussion.  Some  of  the  most  apparent  problems  of  the  OH-58D 
operational  tests  involved  the  lack  of  simulators  or  mock-ups 
for  the  contractor  conducted  factory  training.  This  resulted 
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in  extra  on-site  training  and  could  have  had  a  severe  impact 
on  the  test  results. 


F.  THE  AVENGER  AIR  DEFENSE  SYSTEM 

1 .  Background 

The  Pedestal  Mounted  Stinger,  now  known  as  the 
Avenger,  is  an  air  defense  system  consisting  of  Stinger 
missiles  on  a  turret  mounted  on  the  High  Mobility  Medium 
Wheeled  Vehicle  (HMMWV) .  The  system  went  through  Force 
Development  Testing  and  Experimentation  (FDT&E)  at  Fort 
Hunter- Liggett  in  1989.  The  Initial  Operational  Test  & 
Evaluation  followed  in  1990. 

The  purpose  of  Force  Development  Testing  is  to 
evaluate  and  develop  tactics  for  the  system.  An  advantage  of 
FDT&E  is  that  it  can  be  used  to  evaluate  a  weapon  system  and 
iron  out  'bugs'  prior  to  operational  testing.  Another 
advantage  is  that  it  allows  the  users  to  gain  valuable 
experience  with  the  system.  FDT&E  results  are  not  used  for 
the  Milestone  III  decision  and  give  the  user  an  opportunity 
make  mistakes  and  learn  without  having  the  results  used 
against  the  program. 

2 .  Observations 

During  Avenger  Force  Development  Testing,  exploratory 
trials  were  conducted  for  li  days.  They  had  been  scheduled 
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for  only  one  week,  but  the  sixth  and  final  Avenger  system  had 
not  yet  arrived  from  the  contractor.  Fortunately,  the  testing 
schedule  was  flexible  and  the  extra  days  were  used  for 
additional  platoon  training.  It  also  permitted  Fort  Hunter- 
Liggett  (FHL)  to  improve  its  data  collection  procedures  for 
the  test. 

Only  two  Avenger  firing  units  were  availadDle  for  user 
training  before  the  Force  Development  Test.  Three  more 
arrived  at  FHL  in  time  for  the  tests,  but  as  noted,  the  sixth 
unit  didn't  arrive  until  after  the  scheduled  test  start  date. 
This  meant  that  the  testing  unit  couldn't  conduct  realistic, 
full  platoon  training  until  the  actual  test.  During  previous 
training  at  Fort  Bliss,  the  platoon  had  only  been  able  to  use 
two  available  systems  with  other  surrogate  vehicles. 

When  the  final  Avenger  system  arrived  at  Fort  Hunter- 
Liggett,  it  ceune  directly  from  the  manufacturer.  Because  it 
had  arrived  late,  the  new  sixth  system  wasn't  given  an 
opportunity  to  go  through  an  extensive  mechanical  shakedown  or 
'burn-in'  as  the  previous  systems  had  done.  Since  this  was  a 
Force  Development  Test,  RAM  data  wasn't  recorded  to  determine 
if  this  resulted  in  any  problems.  The  test  report  did  state 
that  "repair  parts  were  not  available"  [Ref.  15] ,  so 
any  break-in  problems  would  have  been  difficult  to  fix. 

Some  difficulties  with  technical  manuals  were  noted 
during  FDT&E.  One  problem  was  that  the  operator  manuals  were 
written  at  a  reading  grade  level  of  11.09  years.  This 
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exceeded  the  Army  Training  and  Doctrine  Command  (TRADOC) 
standard  of  8.0  years,  resulting  in  comprehension  problems  for 
some  operators.  A  similar  but  less  severe  problem  occurred 
with  the  maintenance  manuals  which  were  found  to  be  written  at 
a  grade  level  of  9.65  years.  Both  of  these  problems  were 
subsequently  corrected  prior  to  OT&E. 

Another  problem  was  that  the  Avenger  launch  signature 
device  was  found  to  be  ineffective.  Because  of  this,  the 
firing  of  Stinger  missiles  was  not  discernable  by  attacking 
pilots  or  by  other  Avenger  systems  in  the  test  platoon.  This 
resulted  in  multiple  and  often  unnecessary  engagements  by  the 
weapon  systems. 

A  related  problem  was  that  threat  aircraft  did  not 
have  any  type  of  kill  signature.  The  test  report  stated  that 
in  several  instances,  this  may  have  caused  multiple 
engagements  on  one  aircraft  and  none  on  others. 

The  small  size  and  compartmentalized  terrain  of  Fort 
Hunter-Liggett  limited  testing  scenarios  and  aircraft 
directions  of  attack  to  the  valleys.  This  prevented  the 
Avengers  from  spreading  out  laterally.  It  also  enabled  the 
Avenger  operators  to  focus  their  attention  forward  down  the 
length  of  the  valley.  In  a  more  realistic  operational 
scenario,  the  Avenger  crews  would  be  required  to  keep  360 
degrees  of  observation. 

During  Avenger  operations,  the  driver  is  often 
expected  to  dismount  the  vehicle  as  an  aircraft  observer  while 
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the  gxinner  mans  the  system.  During  Force  Development  Testing, 
the  drivers  indicated  a  need  for  a  method  to  stay  in  voice 
contact  with  the  gunner  while  they  were  away  from  the  vehicle. 
As  soon  as  the  need  was  identified,  a  50  foot  communications 
cable  was  provided  to  the  driver /observer  in  all  systems. 
This  remedied  a  problem  which  probably  would  not  have  been 
identified  until  the  operational  test,  when  it  could  have 
affected  system  performance. 

Finally,  during  Force  Development  Testing,  an  ad  hoc 
cell  from  the  Air  Defense  Artillery  School  and  the  Air  Defense 
Artillery  Board  was  present.  This  cell  was  empowered  to  make 
changes  in  system  utilization  and  tactics  as  they  felt 
necessary.  With  members  of  both  the  school  and  the  board 
present,  many  changes  were  approved  and  recorded  on  the  spot. 
This  enabled  the  Avenger  Platoon  to  get  immediate  feedback  to 
their  suggestions  and  to  quickly  incorporate  the  new  tactics. 

As  a  result  of  FDT&E,  the  Avenger  Program  Manager  was 
able  to  incorporate  many  changes  prior  to  the  system's 
successful  lOT&E  in  1990.  After  the  production  decision  was 
made,  the  system  was  fielded  in  time  to  be  deployed  with  units 
participating  in  Operation  Desert  Shield  and  Operation  Desert 
Storm. 
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O.  AVEMOER  EMHANCENEMT  OB8ERVETZOMS  END  LESSONS  LEESNEO 

1.  T«st  Sehsdulss 

Lesson  Learned.  Tests  should  not  be  scheduled  when  the  tested 
system  may  not  be  available. 

Discussion.  The  Avenger  went  to  the  scheduled  tests  even 
though  all  of  the  systems  had  not  been  delivered  by  the 
contractor  and  the  crews  hadn*t  had  full  platoon  training. 
This  could  have  resulted  in  RAM  problems  and  tactical 
difficulties  due  to  a  lack  of  training  and  experience,  and 
should  be  avoided  in  future  programs. 

2 .  Technical  Manuals 

Enhancement  Observation .  Technical  manuals  were  reviewed  and 
corrected  before  the  operational  test. 

Discussion.  The  Force  Development  Tests  were  used  to  screen 
the  TMs  and  many  comprehension  problems  were  discovered  and 
subsequently  corrected.  If  the  PM  hadn't  gone  through  FDT&E, 
these  problems  might  not  have  been  discovered  until  the 
operational  test,  when  they  could  have  affected  the  RAM  data. 

3 .  Test  Reports 

Enhancement  Observation.  Detailed  test  reports  were  written. 
Discussion.  Comments  from  the  Avenger  FDT&E  were  used  to  make 
system  improvements  prior  to  the  system's  successful 
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operational  test.  Well  written  test  reports  can  provide  the 
PM  with  valuable  information  on  his  system. 


4 .  Test  Resources 

a.  Test  Articles 

Lesson  Learned.  Sufficient  test  articles  should  be  available 
for  the  test. 

Discussion.  The  Avenger  only  had  two  of  the  required  six 
systems  available  for  training  prior  to  FDT&E.  An  the  test 
site,  one  of  the  systems  arrived  late.  This  limited  platoon 
training  time  and  it  could  have  adversely  affected  test 
results . 


b.  Test  Support  Equiiment 

Lesson  Learned.  Sufficient  test  support  equipment  should  be 
available  for  the  test. 

Discussion.  The  Avenger  did  not  have  any  spare  parts  for  the 
conduct  of  this  test.  Any  maintenance  problems  due  to  'burn- 
in'  on  the  new  systems  would  have  negatively  affected  the 
tests.  This  introduced  unnecessary  risk  into  the  program  and 
should  be  avoided  in  future  programs. 

c.  Threat  Systems /Simulators 

Lesson  Learned.  Simulators  should  be  effective. 

Discussion.  During  this  test,  the  simulators  on  the  Avenger 
and  supporting  systems  were  not  very  visible.  This  resulted 


in  unnecessary  engagement  difficulties  which  should  be  avoided 
in  future  programs. 

d.  Vsar  Exparience  Training 

Enhancement  Observation.  Extensive  training  prior  to  OTSE 
gives  the  user  experience  and  can  result  in  feedback  that 
improves  the  system. 

Discussion.  The  Avenger  system  had  a  dedicated  platoon  that 
conducted  extensive  crew  training  at  Fort  Bliss,  Texas  prior 
to  FDT&E.  This  resulted  in  personnel  who  were  knowledgeable 
of  the  system  prior  to  the  test.  During  the  test,  the  users 
were  able  to  make  comments  and  ask  for  improvements,  such  as 
the  50  foot  remote  communications  cable,  that  enhanced  the 
system's  effectiveness.  Because  this  was  a  Force  Development 
Test,  the  discovery  of  problems  did  not  negatively  influence 
the  evaluation  of  the  system  and  it  reduced  program  risk. 

Enhancement  Observation.  Key  decision  makers  and  weapons 
experts  were  present  at  user  training. 

Discussion.  An  advantage  of  this  early  training  was  that  it 
permitted  the  presence  and  participation  of  a  cell  of  Air 
Defense  experts  representing  the  Air  Defense  School  and  Air 
Defense  Board.  They  were  able  to  obtain  direct  user  input  and 
agree  upon  changes  in  utilization  and  tactics  on  the  spot. 
This  enhanced  performance  during  OT&E. 
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H.  THE  ADAT8  AIR  DEFENSE  SYSTEM 


1.  Baokgrotmd 

The  Air  Defense  Anti-tank  System,  also  known  as  the 
Line-of-Site  Forward  Heavy  (I/>S-F-H) ,  is  an  armored  air 
defense  system  that  was  designed  to  operate  at  the  forward 
edge  of  the  battlefield.  In  1986,  Congress  approved  Army 
plans  to  test  the  system  on  a  compressed  time  schedule.  In 
the  fall  of  1987,  the  LOS-F-H  Nondevelopmental  Item  Candidate 
Evaluation  (NDICE)  was  conducted  at  VThite  Sands  Missile  Range. 
Four  candidates  using  off-the-shelf  technology  competed  in  the 
NDICE.  The  ADATS  which  had  been  developed  by  Martin-Marietta, 
was  selected  for  further  development. 

The  ADATS  system  went  through  two  iterations  of  Force 
Development  Testing  and  Experimentation  (FDT&E)  in  1988  and  in 

1989.  The  purpose  of  the  testing  was  to  develop  and  evaluate 
operator,  crew,  squad  and  platoon  tactics  and  drills.  It  was 
also  used  to  evaluate,  modify  as  necessary,  and  validate  the 
Test  Support  Package  (TSP)  for  lOT&E.  This  included  the 
Threat  Support  Package. 

lOT&E  was  conducted  in  two  parts,  the  missile  firing 
phase  and  the  maneuver  phase.  The  test  report  only  describes 
the  maneuver  phase  conducted  by  TEC  at  Fort  Hunter-Liggett 
from  March  through  May  1990.  The  missile  firing  phase  was 
conducted  at  White  Sands  Missile  Range,  New  Mexico  in  February 

1990. 
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2 .  C^servations 


The  ADATS  training  was  conducted  by  the  prime 
contractor.  The  stated  purpose  of  the  training  was  "To  train 
soldiers  on  specific  system  skills  that  are  required  to 
operate  the  system  in  lOT&E".  It  consisted  of  200  hours  of 
instruction  conducted  over  29  days  at  Fort  Bliss,  Texas. 

Prior  to  the  start  of  the  tests,  the  TEXCOM 
Experimentation  Center  conducted  exploratory  trials  (pilot 
tests)  from  27  March  through  5  April.  Fifteen  trials, 
including  one  night  trial,  were  conducted  similar  to  the 
record  trials.  The  objectives  of  the  exploratory  testing 
included  maturation  of  the  data  collection  and  reduction 
procedures  and  resolution  of  instnjmentation  problems.  It 
also  provided  an  opportunity  for  test  controllers  and  players 
to  refine  their  procedures.  In  addition,  the  exploratory 
testing  provided  data  upon  which  to  decide  if  procedures, 
instrumentation,  and  players  were  ready  for  the  record 
trials. [Ref.  16 j 

Fifty  lOT&E  trials  were  conducted  from  9  April  to  23 
May  1990.  Each  trial  was  a  force- on- force  battle  which 
generally  lasted  one  hour.  Normally,  two  trials  were 
conducted  each  day.  Of  the  fifty  trials  conducted,  three  were 
invalidated  due  to  computer,  instrumentation  or  weather 
problems . 

A  number  of  system  manpower  problems  were  discovered 
and  recorded  during  the  test.  The  first  issue  was  that 
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drivers  complained  of  poor  visibility  that  degraded  their 
ability  to  drive  the  vehicle.  Unlike  most  armored  vehicles, 
the  ADATS  didn't  have  a  position  for  a  Track  Commander  (TC)  at 
the  top  of  the  vehicle.  A  vehicle  TC  normally  has  a  better 
field  of  view  and  can  assist  the  driver.  The  lack  of  a  TC  on 
the  ADATS  made  maneuvering  difficult  for  the  driver  whose 
visibility  was  partially  blocked  by  the  vehicle  hull. 

Crew  members  also  complained  of  a  variety  of  physical 
discomforts  inside  of  the  system,  and  expressed  concern  over 
their  immediate  safety,  and  long  term  health  hazards.  Dust 
and  exhaust  seeped  into  the  crew  compartment,  resulting  in 
headaches,  burning  eyes,  and  lung  and  sinus  problems.  Poor 
seating  and  vibration  also  resulted  in  niunbness  in  the 
extremities  and  back  pain.  At  times,  these  problems  cause  the 
crew  to  stop  operations  in  order  to  exit  and  ventilate  the 
vehicle. 

Cramped  conditions  within  the  vehicle  caused  other 
problems.  The  crews  had  to  fit  all  of  their  required  combat 
equipment  into  the  vehicle,  and  this  caused  difficulty  in 
moving  around  Inside.  This  problem  was  aggravated  by  the 
presence  of  test  instrumentation  which  occupied  the  vehicle 
bustle  rack  where  some  of  the  field  gear  would  have  normally 
been  stored. 

Another  interior  space  issue  affected  the  system's 
maintenance  personnel.  They  had  a  difficult  time  replacing 
the  system's  power  distribution  assembly.  The  job  required  two 


personnel,  but  there  was  only  room  for  one  man  near  the 
assembly.  The  test  report  stated  that  the  lack  of  work  space 
and  of  easy  access  to  parts  resulted  in  maintainers  working 
longer  than  would  have  otherwise  been  necessary.  Of  the 
eleven  areas  of  Reliability,  Availability  and  Maintainability 
(RAM)  data  collected,  only  the  Mean  Time  To  Repair  at  the 
organizational  level  met  the  established  requirements. 

The  maneuver  phase  of  the  ADATS  lOT&E  differed 
significantly  from  the  FDT&E  II  which  had  been  conducted  at 
Fort  Bliss.  Some  changes  in  doctrine,  tactics,  techniques, 
and  procedures  resulted  from  lessons  learned  in  FDTE  II.  Other 
differences  occurred  because  of  the  different  focus  of  the 
test,  and  due  to  increased  player  resources  which  were  devoted 
to  the  I0T4E. 

In  the  operational  test,  14  Apache  helicopters  were 
provided  as  surrogates  for  Red  rotary  wing  aircraft,  compared 
to  the  six  provided  for  FDTE  II.  This  continually  gave  the 
required  availability  of  six  operational  aircraft  for  test 
support.  The  remaining  aircraft  were  either  in  maintenance  or 
available  for  pilot  training.  As  a  result,  the  number  of 
threat  aircraft  that  the  ADATS  Platoon  faced  was  substantially 
increased  over  the  previous  force  development  test.  For  the 
operational  test.  Fort  Hunter-Liggett  hired  a  team  of  ADATS 
systems  manpower  evaluators  to  assess  the  causality  of 
soldier/ squad  level  errors.  For  this  analysis,  errors  were 
confined  to  those  resulting  in  non-engagement  of  an  aircraft 
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which  the  crew  should  have  engaged,  and  errors  during  an 
engagement  which  resulted  in  non-intercept  of  the  target.  The 
following  table  categorizes  the  errors: 


Table  1  ERROR  CAUSE  CATEGORY 


Error  Category 

Frequency 

%  of  Total 

Manpower/Personnel/Training 

255 

43.0 

Reliability  &  Maintainability 

226 

38.1  1 

1  Doctrine/Tactics 

80 

13.5  1 

Human  Factors  Engineering 

19 

3.2 

Safety/Health  Hazards 

5 

.8 

Other 

8 

1.4 

TOTAL 

593 

100.0  1 

The  ADATS  operational  testing  concluded  with  the  last 
test  on  23  Nay  1990.  At  Milestone  III,  the  decision  was  made 
not  to  proceed  into  production.  The  primary  reason,  cited  by 
air  defense  experts,  was  loss  of  the  ADATS  primary  mission  due 
to  the  end  of  the  cold  war. 
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I.  ADATS  ENHANCEMENT  OBSERVATIONS  AND  LESSONS  LEARNED 

1 .  Test  Schedules 

Lesson  Learned.  The  operational  testing  schedule  should  be 
based  upon  a  system's  readiness  for  the  test. 

Discussion.  The  ADATS  was  scheduled  for  testing  after  it  had 
gone  through  two  Force  Development  Tests  which  helped  to 
ensure  that  it  was  ready  for  testing.  To  further  ensure  its 
readiness.  Fort  Hunter- Liggett  scheduled  time  for  fifteen 
exploratory  trials  to  ensure  that  the  procedures, 
instrumentation  and  players  were  ready  to  start  the 
operational  test. 

2 .  Test  Reports 

Lesson  Learned.  Test  reports  should  be  detailed. 

Discussion.  The  operational  test  report  for  the  ADATS 
contained  the  fewest  comments  and  the  least  detail  of  the  four 
reports  examined  by  this  thesis.  Some  operator  problems  were 
discussed,  but  not  in  detail.  This  makes  it  difficult  for  the 
Program  Manager  or  other  readers  to  determine  problems  or 
possible  improvements  that  could  be  made. 

Enhancement  Observations.  Information  was  summarized  in  easy 
to  understand  tcdsles. 

Discussion.  The  ADATS  operational  test  report  used  simple 
teUsles  to  summarize  missile  launch  and  error  cause  data.  This 
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method  made  it  much  easier  to  understand  possible  problem 
areas  and  their  sources.  In  other  reports,  the  reader  is 
required  to  search  through  extensive  data  in  order  to  find 
this  information. 

3 .  Test  Resources 

a.  Test  Sites  and  Instrumentation 
Lesson  Learned.  Instrumentation  should  not  affect  user 
performance. 

Discussion.  The  ADATS  crew  members  reported  problems 
operating  inside  of  the  crew/operator  compartment  due  to  the 
instrumentation.  The  instrumentation  had  the  affect  of 
slowing  movement  and  reactions  inside  the  vehicle. 

Jb.  Test  Support  Ecpiipment 

Enhancement  Observation.  Adequate  test  support  equipment  was 
provided  for  the  test. 

Discussion.  For  the  operational  test,  sufficient  Apache 
helicopters  were  provided  to  allow  for  continued  testing 
without  delays  from  maintenance  or  training  problems.  This 
resulted  in  a  full  threat  force  being  available  for  all  tests. 

4.  User  Experience  Training 

Enhancement  Observation.  The  user  platoon  was  well  trained. 
Discussion.  The  ADATS  had  a  dedicated  operational  test 
platoon  for  over  a  year  prior  to  the  test.  This  enabled  the 
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crew  to  receive  extensive  training  in  preparation  for  the 
test.  The  ADATS  platoon  had  200  hours  of  contractor  training 
and  two  Force  Development  Tests  at  Fort  Bliss,  Texas  and  Fort 
Hunter- Liggett  prior  to  OT&E.  This  resulted  in  the  users 
being  well  prepared  for  the  test. 

Enhancement  Observation.  User  training  was  used  to  validate 
test  support  requirements. 

Discussion.  The  ADATS  user  training  was  used  to  validate  the 
test  support  package  for  the  tests.  This  resulted  in 
logistics  support  that  was  able  to  fully  support  the  user  and 
test  requirements. 
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IV 


OPERATIOMAL  TESTING  OBSERVATIONS 


This  thesis  has  determined  that  a  number  of  common 
observations  and  'lessons  learned*  exist  in  the  preparation 
for  and  conduct  of  operational  tests  by  the  Program  Manager. 
These  lessons  Involve  issues  that  hel; ed  the  PM  improve  his 
programs  readiness  for  testing,  as  well  as  those  areas  that 
detracted  from  weapon  system's  readiness.  A  consolidation  and 
categorization  of  these  common  observations  and  ' lessons 
learned'  is  presented  below: 

A.  TEST  SCHEDULES 

Lesson  Learned.  Test  schedules  should  have  some  flexibility 
to  allow  for  delays  caused  by  training,  equipment, 
instrumentation  and  weather  problems. 

Discussion.  Two  out  of  the  four  tests  reviewed  needed 
additional  time  for  activities  other  then  testing  at  the 
beginning  of  the  testing  schedule.  In  one  case,  the  time  was 
gained  by  cramming  the  trials  into  a  shorter  testing  period. 
This  resulted  in  morale  and  support  problems.  In  another 
case,  sufficient  time  had  been  built  into  the  schedule  to 
allow  for  a  delay,  and  no  significant  problems  resulted. 
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Unplanned,  additional  testing  requirements 


should  not  be  added  to  the  test  schedule. 

Discussion.  One  of  the  systems  experienced  a  last  minute 
addition  to  the  testing  requirements.  This  threw  the  schedule 
off  by  several  days  and  resulted  in  information  that  was  of 
little  use.  Once  the  test  schedule  is  established,  additional 
testing  requirements  should  not  be  added  without  increasing 
the  total  test  time. 

Lesson  Learned.  Test  schedules  should  be  established  on  the 
basis  of  system  readiness,  rather  than  strictly  on  milestones. 
Discussion.  Three  out  of  the  four  systems  went  to  the 
operational  test  before  they  were  physically  ready.  This 
caused  problems  that  could  have  resulted  in  system 
cancellation  due  to  poor  test  results.  PMs  shouldn't  test 
their  system  if  it  is  not  ready. 

Lesson  Learned .  Sufficient  time  should  be  planned  in  the 

schedule  for  system  maintenance  and  recovery  after  DT&E. 
Discussion.  One  of  the  four  systems  reviewed  was  adversely 
affected  by  recovery  from  DT&E.  Developmental  testing  is 
designed  to  stress  parts  of  the  weapons  system.  If  OT&E 
follows  too  closely,  the  result  may  be  a  system  that  is  in 
need  of  maintenance  at  the  start  of  the  test.  Even  if  the 
system  is  functional,  there  may  be  a  shortage  of  spare  parts 
due  to  DT&E  recovery  operations.  This  will  seriously  impact 


upon  RAM  data  collection.  In  the  worst  case,  poor  RAM  data 
might  result  in  a  failure  of  the  operational  test. 

B.  TECHNICAL  MANUALS 

Enhancement  Observation.  Contractor  technical  writers  should 
be  brought  to  the  training  and  testing  locations  to  correct 
TMs  as  problems  are  noted  by  the  users. 

Discussion.  Two  of  the  four  system  test  reports  addressed  TM 
problems.  In  both  cases,  these  errors  were  caught  and 
corrected  before  OT&E.  These  errors  can  affect  system 
operation  and  maintenance  support.  Research  for  this  thesis 
indicates  that  the  most  practical  way  to  screen  and  fix  TM's 
is  to  involve  the  contractors  technical  writers  early  in  the 
users  training  and  testing.  This  early  screening  will  help  to 
ensure  accuracy  and  clarity.  OT&E  is  the  wrong  time  to 
discover  that  the  manuals  are  wrong  or  unclear,  but  if  they 
are,  having  the  technical  writers  present  will  help  ensure 
that  the  problems  are  fixed  on  the  spot. 

C.  TEST  REPORTS 

Lesson  Learned.  Test  reports  should  give  a  detailed  report  of 
what  actually  happened  in  the  test. 
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Discussion,  Only  one  of  the  four  test  reports  gave  sufficient 
details  of  what  took  place  during  the  tests.  Test  reports 
need  to  tell  why  things  happened  the  way  they  did.  The 
evaluator  needs  to  report  more  then  just  events.  For  tests  in 
which  the  human  factors  are  involved,  the  most  neglected 
resource  in  finding  out  why  things  happened  the  way  they  did 
is  the  test  participants  themselves.  The  tester  and  the  PM 
should  not  let  them  go  without  debriefing  them  extensively  and 
recording  their  explanations  of  why  things  happened  the  way 
they  did. [Ref.  17] 

D.  TEST  RESOURCES 

1.  Test  Articles 

Lesson  Learned.  Sufficient  test  articles  should  be  produced 
and  available  well  before  the  operational  test  is  supposed  to 
start . 

Discussion.  In  three  of  the  cases  reviewed,  the  lack  of 
complete  test  articles  caused  truncated  and  ineffective 
training  prior  to  OT&E.  In  one  case,  the  users  and  supporters 
had  minimal  hands  on  experience  with  the  system  before  the 
operational  test.  In  another,  the  testing  had  to  be  delayed 
for  almost  a  week  to  permit  the  delivery  of  all  of  the 
required  vehicles  for  testing.  In  a  third,  the  users  didn't 
have  a  fully  functioning  systems  due  to  subsystem  integration 
problems . 
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Lesson  Learned.  Systems  should  go  through  burn-in  prior  to 
the  operational  test. 

Discussion.  In  one  system  test,  the  final  weapon  system 
literally  arrived  off  of  the  truck  from  the  factory.  While 
this  did  not  appear  to  have  affected  the  test  results,  it  did 
introduce  unnecessary  risk  into  the  program.  When  mean  time 
between  failures  (MTBF)  is  to  be  tested,  it  is  better  to  test 
equipment  that  has  already  been  through  its  burn-in  period. 

2.  Test  Sites  and  Instrumentation 
Lesson  Learned.  Test  sites  should  be  adequate  in  size,  and 
all  special  clearances  should  be  obtained. 

Discussion.  This  is  especially  important  when  using  devices 
such  as  laser  range  finders  and  certain  kinds  of 
communications  equipment.  These  special  items  frequently 
require  coordination  with  outside  agencies  such  as  the  FAA  or 
the  Forest  Service.  A  ten  day  delay  was  experienced  by  the 
OH-58D  when  the  system  was  prevented  from  using  its  laser 
range  finder  during  scheduled  testing. 

Lesson  Learned.  Test  instrumentation  should  not  interfere 
with  user  operations. 

Discussion.  In  one  of  the  four  tests,  this  was  a  problem. 
Instrumentation  can  be  bulky  and  interfere  with  movements  of 
the  users.  Instrumentation  should  provide  the  needed  data. 
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but  not  hinder  the  users.  It  should  be  as  unobtrusive  as 
possible  to  the  test  participants. 

3.  Test  Support  Equlpnent 

Lesson  Learned.  All  necessary  support  equipment  should  be 
available  and  operable. 

Discussion.  In  two  of  the  four  systems  examined,  the  Program 
Manager  had  extra  support  equipment  available  as  back-ups 
during  the  test.  If  the  test  involves  a  baseline  comparison 
with  another  system,  or  is  supported  by  another  system,  make 
sure  that  appropriate  emphasis  is  also  placed  on  that  system's 
availability.  These  extra  systems  assured  continuous  support 
and  gave  the  operators  extra  training  systems. 

4 .  Threat  Systeas/Simulators 

Lesson  Learned.  Threat  systems  should  actually  look  like  the 
threat  systems  and  not  the  friendly  system  they  were  derived 
from. 

Discussion.  This  problem  occurred  in  one  of  the  tests 
reviewed.  Poor  threat  systems  can  cause  confusion  among 
personnel  who  are  trying  to  identify  targets.  This  confusion 
can  result  in  of  fratricide  and  failures  to  engage  enemy 
vehicles.  These  kinds  of  actions  result  in  data  that  make  the 
weapon  system  appear  ineffective. 
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Lesson  Learned.  Threat  system  crew  members  should  fully 
understand  the  threat  tactics  they  are  supposed  to  use. 
Discussion.  In  some  tests,  improper  tactics  resulted  in 
invalidated  test  results.  Training  for  test  support  personnel 
such  as  threat  crew  members  requires  emphasis  and  verification 
before  the  test  begins. 

Lesson  Learned .  All  weapons  effects  simulators  should  be 
tested  and  judged  to  be  realistic  and  effective  prior  to  the 
tests . 

Discussion.  There  was  a  problem  with  simulators  in  all  four 
system  tests.  Ineffective  kill  lights  and  flash/bang  devices 
result  in  wasted  rounds  and  unnecessary  engagements  from  the 
tested  weapon  systems.  In  a  realistic  combat  environment, 
crew  members  rely  visual  clues  to  determine  the  effectiveness 
of  engagements.  These  visual  clues  are  just  as  important  in 
the  operational  test.  A  lack  of  preparation  in  this  area  can 
result  in  quantatative  data  that  make  the  weapon  system  look 
ineffective. 

5.  Operational  Force  Test  Support 

Lesson  Learned.  Detailed  memorandums  of  understanding  (MOUs) 
should  be  executed  with  all  military  units  providing  test 
support  personnel. 
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Discussion.  One  of  the  four  tests  addressed  this  as  an  area 
of  en^hasis  due  to  misunderstandings  with  the  test  support 
unit.  Fort  Hunter -Liggett  testers  believe  that  conflict 
resolution  is  a  normal  part  of  their  mission,  but  testing 
could  be  iirqproved  if  the  test  support  units  had  a  more 
conplete  understanding  of  the  requirements  and  training  that 
they  must  corr5)lete  prior  to  OT&E.  Failure  to  complete  this 
requisite  training  results  in  support  personnel  who  are  not 
qualified  and  must  be  retrained.  This  uses  up  valuable  time 
at  the  beginning  of  the  test  and  could  possibly  endanger  the 
test  itself. 

Lesson  Learned .  Support  personnel  should  always  be  fully 
informed  of  the  latest  requirements  and  changes. 

Discussion.  The  test  support  units  need  to  fully  understand 
the  requirements  of  the  test  so  that  their  questions  and 
objections  may  be  satisfied  well  in  advance.  The  supporting 
units  often  have  legitimate  questions  involving  issues  such  as 
safety  which  should  not  come  up  just  prior  to  test  execution. 
These  last  minute  conflicts  can  be  reduced  with  support  unit 
involvement  and  understanding. 

Lesson  Learned.  Contractor  training  should  be  observed  and 
validated. 

Discussion.  In  one  of  the  programs,  contractor  training  was 
ineffective  and  incomplete  due  to  a  lack  of  training  assets 
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and  funding.  The  PM  needs  to  be  assured  that  the  contractor 
training  is  up  to  an  agreed  upon  standard  and  that  it  is 
consisted  when  these  standards  have  been  met.  This  training 
needs  to  include  sufficient  hands-on  training  time  on  a 
representative  system,  not  just  mock-ups.  If  the  PM 
recognizes  that  contractor  training  is  insufficient,  he  needs 
to  plan  to  correct  this  shortfall  within  the  testing  schedule. 

Lesson  Learned.  OT&E  should  be  restricted  to  a  reasonable 
duration. 

Discussion.  Many  test  support  personnel  come  from  other 
military  facilities  for  the  duration  of  the  test.  Overly 
lengthy  tests  can  result  in  morale  problems  that  may  impact  on 
test  results.  According  to  Ft.  Hunter -Liggett  testers,  this 
is  especially  true  when  the  tests  encompass  major  family 
oriented  holidays.  Tests  are  too  important  to  risk  adverse 
results  from  tired  or  apathetic  soldiers. 


B.  USER  EXPERIENCE  TRAINING 

Lesson  Learned.  The  Program  Manager  should  schedule  Force 
Development  testing  and  training  prior  to  lOT&E. 

Discussion.  Two  of  the  systems  reviewed  made  extensive  use  of 
Force  Development  Tests  before  lOT&E.  This  testing  resulted 
in  inqportant  user  familiarization  and  problem  identification 
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and  resolution  prior  to  the  Operational  Test.  While  the 
primary  purpose  of  FDTStE  is  to  verify  tactics  and  crew  drills, 
it  also  allows  users  to  interact  with  and  learn  the  system. 
This  helps  the  users  to  find  problems  and  make  suggestions  for 
system  inprovements  before  the  operational  test,  when  those 
problems  could  negatively  affect  the  systems  evaluation. 

Lesson  Learned.  Training  should  not  be  conducted  too  early, 
since  there  may  not  be  sufficient  production  representative 
systems  available  to  support  the  training,  and  users  may 
forget  the  training. 

Discussion.  In  one  system,  the  training  was  conducted  several 
months  too  early.  This  required  extensive  retraining  that  ate 
into  the  testing  schedule  and  caused  other  testing 
repercussions. 

Lesson  Learned.  Prototypes  or  detailed  mock-ups  need  to  be 
available  for  all  training  conducted  before  OT&E. 

Discussion.  Only  two  of  the  systems  examined  had  sufficient 
prototypes  to  train  on  before  the  test.  There  should  be  a 
sufficient  number  availeible  to  accommodate  all  personnel  in 

I 

training  before  the  operational  test. 
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V.  C0MCLUSI0M8  AMD  RBCOMNEMDATZOM8 


A.  C0NCLD8I0M8 

Because  testing  is  a  major  cost  and  schedule  driver, 
adequate  planning  is  essential  long  before  the  start  of  any 
testing.  Test  planning  and  continuous  coordination  between 
the  Program  Manager,  the  operational  tester  and  the  contractor 
is  essential  to  the  success  of  the  weapon  system  during  OT&E. 

DOD  decisionmakers  rely  on  the  results  of  operational 
testing  to  estimate  weapon  system  performance.  But,  in  the 
systems  this  thesis  reviewed,  problems  occurred  which  may  have 
adversely  affected  the  performance  which  the  production 
decision  would  have  been  based  upon.  For  this  reason,  the  PM 
needs  to  ensure  that  extensive  effort  has  been  made  in  the 
actual  preparation  for  OT&E. 

In  the  earliest  tests  examined  for  this  thesis,  allowances 
were  made  for  errors  and  limited  success  was  acceptable  in 
OT&E.  Weaknesses  in  the  weapon  systems  seemed  to  be  treated 
as  something  which  could  be  fixed  after  the  test.  Today, 
these  weaknesses  would  most  likely  result  in  cancellation  of 
the  program.  Program  managers  need  to  place  greater  emphasis 
in  the  areas  of  test  schedules,  manuals,  reports,  and 
training! 
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B.  BPECZriC  CONCLUSIONS 

The  following  specific  conclusions  were  derived  from  the 
analysis  of  these  four  programs: 

1.  Test  Schedules 

This  thesis  has  concluded  that  pre-established 
schedules  are  driving  the  tests,  not  system  readiness. 
Instead  of  testing  a  system  when  it  is  ready,  the  tendency  is 
to  test  the  system  when  it  is  scheduled.  The  PM's  goal  is  to 
field  an  operationally  effective  weapon  system,  and  this  is 
not  always  compatible  with  meeting  the  schedule.  In  the  four 
systems  examined.  Program  Managers  have  generally  not  ensured 
that  their  systems  were  fully  configured  and  ready  for 
operational  testing. 

2 .  Technical  Manuals 

This  thesis  concludes  that  early  attention  to 
technical  manuals  resulted  in  a  more  accurate  product  and  led 
to  fewer  logistics  support  problems  during  OT&E.  TMs  are  an 
integral  part  of  system/ equipment  support  requirements.  They 
are  the  prime  means  of  communicating  maintenance  and 
operational  information  to  the  user.  Since  the  quality  of  TMs 
affects  equipment  maintainability,  personnel  efficiency, 
safety  and  readiness,  quality  in  TMs  should  always  be  a 
planned  objective. [Ref .  18]  It  is  imperative  that  all 
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system  publications  and  manuals  be  completed,  reviewed  and 
selectively  tested  prior  to  the  beginning  of  operational 
testing. 


3 .  Test  Reports 

This  thesis  concludes  that  operational  test  reports 
lack  consistency  and  completeness  in  their  depth  of  coverage. 
This  weaknesses  leads  to  reports  that  do  not  clearly  report 
what  happened  in  the  test.  The  reports  are  not  as  useful  to 
decisionmakers  or  PMs  as  they  could  be. 

4 .  Test  Resources 

This  thesis  concludes  that  the  majority  of  the 
problems  which  occurred  during  OT&E  are  directly  related  to 
test  resource  issues.  Part  V.  of  the  TEMP  details  the 
resources  required,  however  they  do  not  appear  to  get  the 
attention  that  they  warrant.  The  GAO  stated  that  "Common 
weaknesses  in  the  quality  of  such  testing  that  we  have 
reported  include  the  lack  of  realism,  independence,  and  test 
resources  in  the  planning,  execution,  and  evaluation  of  the 
tests. [Ref.  19]  They  cited  twenty-seven  cases  where 
important  test  resources  were  limited  or  not  available  for 
testing. [Ref .  20]  In  spite  of  this  apparent  history 
of  problems,  resources  still  do  not  get  the  attention  they 
deserve . 
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5.  U««r  Bspariane*  Training 


The  final  conclusion  of  this  thesis  is  that  user 
experience  training  and  testing  before  the  operational  test  is 
extremely  valuable  to  the  Program  Manager  and  his  system. 
This  training  helps  to  ensure  that  problems  are  discovered 
before  the  test.  It  is  also  the  best  way  for  the  users  to 
gain  realistic  experience  with  the  system  before  they  are 
evaluated.  The  sooner  the  user  is  exposed  to  the  system,  the 
better  things  will  go  during  OT&E. 


C.  RECOMMEMDATIOMS 

To  irprove  the  operational  testing  process,  this  thesis 
recommends  that  Program  Managers  and  testers  review  and 
address  the  most  comiaon  issues  that  have  affected  systems  that 
have  already  gone  through  testing.  The  Assistant  Program 
Manager  (APM)  for  testing  should  address  the  specific  issues 
of  schedules,  technical  manuals,  resources  and  user  training. 
These  are  the  building  blocks  of  a  successful  operational 
test. 

The  list  of  'lessons  learned*  detailed  in  this  thesis  are 
an  important  tool  which  can  give  the  PM  an  understanding  of 
potential  problem  areas  and  how  to  avoid  or  overcome  them. 
Program  Manager  Office  personnel  and  testers  should  review 
this  list  before  evaluating  system  readiness  for  testing.  In 
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addition  they  should  keep  an  active  list  of  lessons  learned 
for  future  Program  Managers  and  test  personnel. 

To  improve  the  operational  testing  process,  this  thesis 
makes  the  following  specific  recommendations: 


1 .  Test  Schedules 

•  Test  schedules  should  have  some  flexibility  to  allow  for 
delays  caused  by  training,  equipment,  instrumentation  and 
weather  problems. 

•  Unplanned,  additional  testing  requirements  should  not  be 
added  to  the  test  schedule. 

•  Test  schedules  should  be  established  on  the  basis  of 
systems  readiness,  rather  than  strictly  on  milestones. 

•  Sufficient  time  should  be  planned  in  the  schedule  for 
system  maintenance  and  recovery  after  DT&E. 


2 .  Technical  Manuals 

•  Contractor  technical  writers  should  be  brought  to  the 
training  and  testing  locations  to  correct  TM’s  as  problems 
are  noted  by  the  users. 


3 .  Test  Reports 

•  Test  reports  should  give  a  detailed  report  of  what 
actually  happened  in  the  test. 


4 .  Test  Resources 
a.  Test  Articles 

•  Sufficient  test  articles  should  be  produced  and  available 
well  before  the  operational  test  is  supposed  to  start. 
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Systems  should  go  through  burn-in  prior  to  the  operational 
test. 


b.  Test  Sites  and  Instrumentation 

Test  sites  should  be  adequate  in  size,  and  all  special 
clearances  should  be  obtained. 

Test  instrumentation  should  not  interfere  with  user 
operations. 


c.  Test  Support  Equipment 

All  necessary  support  equipment  should  be  available  and 
operable . 


d.  Threat  Systems /Simulators 

Threat  systems  should  actually  look  like  the  real  threat 
systems  and  not  the  friendly  system  they  were  derived 
from. 

Threat  system  crew  members  should  fully  understand  the 
threat  tactics  they  are  supposed  to  use. 

All  weapons  effects  simulators  should  be  tested  and  judged 
to  be  realistic  and  effective  prior  to  the  tests. 


e.  Operational  Force  Test  Support 

Detailed  memorandums  of  understanding  (MOUs)  should  be 
executed  with  all  military  units  providing  test  support 
personnel . 

Support  personnel  should  always  be  fully  informed  of  the 
latest  requirements  and  changes. 

Contractor  training  should  be  observed  and  validated. 
OT&E  should  be  restricted  to  a  reasonable  duration. 


5.  ns«r  Bxperl«nc«  Training 


•  The  Program  Manager  should  schedule  Force  Development 
testing  and  training  prior  to  lOT&E. 

•  Training  should  not  be  conducted  too  early,  since  there 
may  not  be  sufficient  production  representative  systems 
availed)le  to  support  the  training,  and  users  may  forget 
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AAH 

AH 

ADATS 

ADM 

AHIP 

APM 

ARTEP 

ASA(RDA) 

ASARC 

DA 

DAB 

DOD 

DODI 

DOT&E 

DSMC 

DT 

DT&E 

EMD 

FDT 

FDT&E 


Advanced  Attack  Helicopter 

Attack  Helicopter 

Air  Defense  Anti-Tank  System 

Acquisition  Decision  Memorandiam 

Advanced  Helicopter  Improvement  Program 

Assistant  Progreun  Manager 

Army  Training  Evaluation  Program 

Assistant  Secretary  of  the  Army  (Research, 

Development  &  Acquisition) 

Army  Systems  Acquisition  Review  Council 

Department  of  the  Army 

Defense  Acquisition  Board 

Department  of  Defense 

Department  of  Defense  Instruction 

Director  of  Operational  Test  &  Evaluation 

Defense  Systems  Management  College 

Development  Test 

Developmental  Test  &  Evaluation 

Engineering  and  Manufacturing  Development 

Force  Development  Testing 

Force  Development  Test  &  Experimentation 

Folding  Fin  Aerial  Rocket 


FFAR 


FHL 

FOT&E 

GAO 

ILS 

IOC 

lOT&E 

IPS 

LCC 

MAA 

MNS 

MOU 

MPT 

MTTR 

OH 

OPEVAL 

ORD 

OTA 

OT  I 

OT&E 

PEO 

PM 

PPBS 

P3I 

R&D 

RAM 

SECFDEF 


Fort  Hunter -Ligget 

Follow-on  Operational  Test  and  Evaluation 

General  Accounting  Office 

Integrated  Logistics  Support 

Initial  Operational  Capability 

Initial  Operational  Test  &  Evaluation 

Integrated  Program  Summary 

Life  Cycle  Cost 

Mission  Area  Analysis 

Mission  Need  Statement 

Memorandum  of  Understanding 

Manpower,  Personnel  and  Training 

Mean  Time  To  Repair 

Observation  Helicopter 

Operational  Evaluation 

Operational  Requirements  Docximent 

Operational  Test  Agency 

Initial  Operational  Test 

Operational  Test  &  Evaluation 

Program  Executive  Officer 

Program  Manager/Project  Manager/ Product  Manager 
Planning,  Progreunming  and  Budgeting  System 
Pre-Planned,  Product  Improvement 
Research  and  Development 

Reliability,  Availability,  and  Maintainadiility 
Secretary  of  Defense 
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T&E  Test  and  Evaluation 

TADS  Target  Acquisition  Designation  Sight 

TADSS  Training  Aids,  Devices,  Simulators  and  Simulations 

TEC  TEXCOM  Experimentation  Center 

TEMP  Test  and  Evaluation  Master  Plan 

TM  Technical  Manual 

TMP  Technical  Manual  Plan 

TSP  Test  Support  Package 

WBS  Work  Breakdown  Structure 
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APPENDIX  Bt  GLOSSARY  OP  DEFINITIONS 


ACQUISITION  -  The  process  consisting  of  planning,  designing, 
producing,  and  distributing  a  weapon  system/equipment. 
Acquisition  in  this  sense  includes  the  conceptual,  validation, 
full  scale  development,  production,  and  deployment/operational 
phases  of  the  weapon  systems/equipment  project.  For  those 
weapons  systems  not  being  procured  by  a  project  manager,  it 
encompasses  the  entire  process  from  inception  of  the 
rec[uirement  through  the  operational  phase. 

ACQUISITION  DECISION  MEMORANDUM  -  A  memorandum  signed  by  the 
milestone  decision  authority  that  documents  decisions  made  and 
the  exit  criteria  established  as  the  result  of  a  milestone 
decision  review  or  in-process  review. 

ADVANCED  DEVELOPMENT  -  Includes  all  projects  which  have  moved 
into  the  development  of  hardware  for  tests. 

ANALYSIS -The  qualitative  and/or  quantified  evaluation  of 
information  requiring  technical  knowledge  and  judgement. 
AVAILABILITY  -  A  measure  of  the  degree  to  which  cui  item  is  in 
em  operable  and  commit table  state  at  the  start  of  a  mission, 
when  the  mission  is  called  for  at  an  unknown  (random)  time. 
BRASSBOARD  -  An  experimental  device  (or  group  of  devices)  used 
to  determine  feasibility  and  to  determine  teclinical  and 
operational  data.  It  normally  will  be  a  model  sufficiently 
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hardened  to  outside  of  environments  to  demonstrate  the 
technical  and  operational  properties  of  immediate  interest. 
It  may  resemble  the  end  item,  but  is  not  intended  for  use  as 
the  end  item. 

BREADBOARD  -  An  experimental  device  (or  group  of  devices)  used 
to  determine  feasibility  and  to  develop  technical  data.  It 
normally  will  be  configured  only  for  laboratory  use  to 
demonstrate  the  technical  principles  of  immediate  interest. 
It  may  not  resemble  the  end  item  and  is  not  intended  for  use 
as  the  projected  end  item. 

DEVELOPMENT  TEST  &  EVALUATION  -  That  test  and  evaluation 
conducted  to  assist  the  engineering  design  and  development 
process  and  to  verify  attainment  of  technical  performance 
specifications  and  objectives. 

EFFECTIVENESS  -  The  performance  or  output  received  from  an 
approach  or  a  prograun.  Ideally,  it  is  a  quantitative  measure 
which  can  be  used  to  evaluate  the  level  of  performance  in 
relation  to  some  standard,  set  of  criteria,  or  end  objective. 
FOLLOW-ON  OPERATIONAL  TEST  AND  EVALUATION  (FOT&E)  -  All  OT&E 
after  the  Production  and  Deployment  Decision. 

INITIAL  OPERATIONAL  TEST  AND  EVALUATION  (lOT&E)  -  All  OT&E 
prior  to  the  Production  and  Deployment  Decision. 

INTEGRATED  PROGRAM  SUMMARY  -  A  DoD  Component  document  prepared 
and  submitted  to  the  milestone  decision  authority  in  support 
of  Milestone  I,  II,  III,  and  IV  reviews.  It  provides  an 
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independent  assessment  of  a  program's  status  and  readiness  to 
proceed  into  the  next  phase  of  the  acquisition  cycle. 
INTEROPERABILITY  -  The  cUaility  of  systems,  units,  or  forces  to 
provide  services  to,  and  accept  services  from,  other  systems, 
units  or  forces,  and  to  use  the  services  so  exchanged  to 
encUsle  them  to  operate  together  effectively. 

LIFE  CYCLE  COST  -  The  total  cost  to  the  Government  for  the 
development,  acquisition,  operation  and  logistic  support  of  a 
system  or  set  of  forces  over  a  defined  life  span. 

LOGISTICS  SUPPORT  -  The  supply  and  maintenance  of  material 
essential  to  proper  operation  of  a  system  in  the  force. 
LOGISTICS  SUPPORTABILITY  -  The  degree  to  which  the  planned 
logistics  (including  test  equipment,  spares  and  repair  parts, 
technical  data,  support  facilities,  and  training)  and  manpower 
meet  system  availaibility  and  wartime  usage  requirements. 
MAJOR  SYSTEM  ACQUISITION  •  A  system  acquisition  program 
designated  by  the  SECDEF  to  be  of  such  importance  and  priority 
as  to  require  special  management  and  attention. 

MISSION  NEED  STATEMENT  -  A  non- system  specific  statement  of 
operational  capeibility  need,  prepared  lAW  the  format  in  DoD 
5000. 2-M. 

OPERABILITY  -  The  design  characteristic  of  the 
system/equipment  that  will  assure  personnel  feasibility  and 
optimum  utilization  of  operator  personnel. 

OPERATIONAL  AVAILABILITY  (AO)  -  An  index  of  a  weapon  system's 
material  readiness,  including  system  software  where 
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appliccUole,  in  a  mission  environment.  It  is  a  measure  of  the 
probability  of  an  items  being  in  a  condition,  generally 
referred  to  as  "up",  such  that  is  can  perform  its  intended 
function,  within  acceptable  limits  of  degradation,  when  called 
upon. 

OPERATIONAL  EFFECTIVENESS  -  The  capability  of  the  system  to 
perform  its  intended  function  effectively  over  the  expected 
range  of  operational  circumstances,  in  the  expected 
environment,  and  in  the  face  of  the  expected  threat,  including 
countermeasures . 

OPERATIONAL  REQUIREMENT  -  The  basic  requirement  document  for 
all  DoD  acquisition  programs  requiring  research  and 
development  effort. 

OPERATIONAL  SUITABILITY  -  The  capability  of  the  system,  when 
operated  and  maintained  by  typical  military  users  in  the 
expected  numbers  and  of  the  expected  experience  level,  to  be 
reliable,  maintainable,  operationally  available,  logistically 
supportedJle  when  deployed,  compatible,  and  interoperable. 
OPERATIONAL  TEST  AND  EVALUATION  -  The  field  test  under 
realistic  combat  conditions,  of  any  item  (or  key  component  of) 
weapons,  ec[uipment,  or  munitions  for  the  purpose  of 
determining  the  effectiveness  and  suitability  of  the  weapons, 
equipment,  or  munitions  for  use  in  combat  by  typical  military 
users,  and  the  evaluation  of  the  results  of  such  test. 
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PREPRODUCTION  PROTOTYPE  -  An  article  in  final  form  employing 
standard  parts,  representative  of  articles  to  be  produced 
siobseguently  in  a  production  line. 

PRODUCTION  AND  DEPLOYMENT  DECISION  -  The  Milestone  III 
decision  by  which  the  SECDEF  reaffirms  the  mission  need, 
confirms  the  system  as  ready  for  production,  approves  the 
system  for  production,  and  authorizes  the  Component  to  deploy 
the  system  to  the  using  activity. 

PROGRAM  -  A  plan  or  scheme  of  action  designed  for  the 
accomplishment  of  a  definite  objective  which  is  specific  as  to 
the  time -phasing  of  the  work  to  be  done  and  the  means  proposed 
for  its  accomplishment,  particularly  in  quantitative  terms, 
with  respect  to  manpower,  material,  and  facilities 
requirements . 

PROGRAM  MANAGER  -  The  individual  in  the  DoD  who  manages  a 
major  system  acquisition  program. 

PROGRAM  OBJECTIVES  MEMORANDUM  -  A  biennial  memorandum  in 
prescribed  format  submitted  to  SECDEF  in  April  by  the  DoD 
components  head  which  recommends  the  total  resource 
requirements  and  prograuns  within  the  parameters  of  SECDEF' s 
fiscal  guidance. 

RELIABILITY  -  The  probcdaility  that  an  item  will  perform  its 
intended  functions  for  a  specified  period  of  time  under  stated 
conditions . 

RELIABILITY,  AVAILABILITY,  and  MAINTAINABILITY  (RAM) 
Requirement  imposed  on  acquisition  systems  to  ensure  they  are 


84 


operationally  ready  for  use  when  needed,  will  successfully 
perform  assigned  functions,  and  can  be  economically  operated 
and  maintained  within  the  scope  of  logistics  concepts  and 
policies. 

REQUIRED  OPERATIONAL  CAPABILITY  (ROC)  -  A  brief  statement  of 
a  specific  operational  capability  which  is  required  in  the 
mid-range  period. 

SURVIVABILITY  -  The  degree  to  which  a  system  is  able  to  avoid 
or  withstand  a  hostile  environment  without  suffering  an 
abortive  impairment  of  its  ability  to  accomplish  its 
designated  mission. 

TEST  CRITERIA  -  Standards  by  which  test  results  and  outcome 
are  judged. 

TEST  AND  EVALUATION  -  Process  by  which  a  system  or  components 
are  compared  against  requirements  and  specifications  through 
testing.  The  results  are  evaluated  to  assess  progress  of 
design,  performance,  supportability,  etc.  There  are  three 
types  of  T&E  -  Developmental  (DT&E) ,  Operational  (OT&E) ,  and 
Production  Acceptance  (PAT&E)  -  occurring  during  the 
Acquisition  cycle. 

THREAT  -  The  sum  of  the  potential  strength,  capabilities,  and 
intentions  of  an  enemy  which  can  limit  or  negate  mission 
accoir^lishment  or  reduce  force,  system,  or  equipment 
effectiveness . 
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TRANSPORTABILITY  -  The  inherent  capability  of  material  to  be 
moved  by  towing,  by  self -propulsion,  or  by  carrier  via 
railways,  highways,  waterways,  pipelines,  ocean,  and  airways. 
VULNERABILITY  -  The  characteristics  of  a  system  which  causes 
it  to  suffer  a  definite  degradation  as  a  result  of  having  been 
subjected  to  a  certain  level  of  effects  in  a  man-made  hostile 
environment . 


86 


LIST  OF  REFERENCES 


1. Defense  Systems  Management  College,  Systems  Engineering 
Management  Guide.  DSMC,  Fort  Belvoir,  VA,  [January  1990] ,  p. 
13-1. 


2. D0D  INSTRUCTION  5000.2,  Defense  Accruisition  Management 

Policies  and  Procedures.  [February  23,  1991],  p.  2-1. 

3.  Defense  Systems  Management  College,  Systems  Engineering 
Management  Guide.  DSMC,  Fort  Belvoir,  VA,  [January  1990] , 

p.  13-1. 

4. D0D  INSTRUCTION  5000.2,  Defense  Acquisition  Management 

Policies  and  Procedures.  [February  23,  1991],  p.  8-2. 

5. D0D  INSTRUCTION  5000.2,  Defense  Acquisition  Management 

Policies  and  Procedures.  [February  23,  1991],  p.  8-6. 

6.  Defense  Systems  Management  College,  Systems  Engineering  and 
Management  Guide.  DSMC,  Fort  Belvoir,  VA,  [January  1990] ,  p. 
13-16. 

7. Ibid,  p.  13-16. 

8. D0D  5000. 2 -M,  Defense  Accruisition  Management  Documentation 
and  Reports.  [February  23,  1991],  p.  7-1-10. 

9.  Defense  Systems  Management  College,  The  Program  Managers 
Notebook.  [June  1992],  p.  6.1-3. 

10. Ibid. .  p.  5.2-2. 

11. Ibid. .  p.  5.3-1. 

12.  Advanced  Attack  Helicopter  Operational  Test  II,  FTR  OT-046 
[April  1982],  p.  474. 

13.  GAO,  APACHE  HELICOPTER:  Test  Results  for  30-Millimeter 
Weapon  System  Inconclusive.  GAO/NSIAD-93-134  [April  1993]. 

14.  Advanced  Helicopter  Improvement  Program,  Operational  Test 
II  [February  1985],  p.  3-21. 

15.  Avenger  Force  Development  and  Experimentation  Test,  Phase 
II  [August  1989],  p.  2-125. 


87 


16. Line  of  Sight -Forward  Heavy,  Initial  Operational  Test 
[August  1990],  p.  2-3. 


17.  Dr.  Ernest  A.  Seglie,  The  Elements  of  Test  &  Evaluation, 
[n.d.  ]  . 

18.  The  Defense  Systems  Management  College,  The  Progrcun 
Managers  Notebook.  [June  1992],  p.  5.3-1. 

19.  General  Accounting  Office,  WEAPONS  ACQUISITION;  A  Rare 
Opportunity  for  Lasting  Change.  GAO/NSIAD-93-15,  [December 
1992] ,  p.  27. 

20.  GAO,  WEAPON  PERFORMANCE:  Operational  Test  and  Evaluation 
Can  Contribute  More  to  Decisionmaking.  GAO/NSIAD-87-57, 
[December  1986],  p.  14. 


88 


BIBLIOGRAPHY 


Blanchard,  Benjamin,  Logistics  Engineering  And  Management. 
Fourth  Edition,  New  Jersey:  Prentice  Hall,  1992. 

Blanchard,  Benjamin,  Systems  Engineering  And  Management.  New 
Jersey:  Prentice  Hall,  1992. 

General  Accounting  Office,  Weapon  Performance:  Operational 
Test  and  Evaluation  Can  Contribute  More  to  Decisionmaking. 
GAO/NSIAD-87-57,  December  23,  1986. 

General  Accounting  Office,  Naw  Weapons  Testing:  Defense 
Policy  on  Early  Operational  Testing.  GAO/NSIAD-89-98,  U.S. 
Government  Printing  Office,  May  8,  1989. 

General  Accounting  Office,  Teat  And  Evaluation:  An  Assessment 

_ POP  *6 _ Oper^tiop^l _ Tgst _ and  Evaluation  Capability 

Improvement  Program.  GAO/NSIAD-90-141,  March  30,  1990. 

General  Accounting  Office,  Weapons  Testing:  POD  Needs  to  Plan 
and  Conduct  More  Timely  Operational  Tests  and  Evaluation. 
GAO/NSIAD-90-107,  May  17,  1990. 

General  Accounting  Office,  Weapons  Testing:  Concurrency  in  the 
Acquisition  Process.  GAO/T-NSIAD-90-43 ,  May  17,  1990. 

General  Accounting  Office,  Apache  Helicopter:  Test  Results  for 
30 -Millimeter  Weapon  System  Inconclusive.  GAO/NSIAD-93-134, 
April  1,  1993. 

Dr.  Ernest  A.  Seglie,  "The  Ever  Current  Issues  In  OT&E, " 
Program  Manager.  September- October  1993. 

Dr.  Ernest  A.  Seglie,  "The  Elements  of  Test  &  Evaluation," 
[n.d. ]  . 


Defense  Systems  Management  College,  Lessons  Learned;  Multiple 
Launch  Rocket  System.  Fort  Belvoir,  VA. ,  July  1980. 

Defense  Systems  Management  College,  Lessons  Learned;  Advanced 
Attack  Helicopter.  Fort  Belvoir,  VA. ,  July  1983. 

The  Defense  Systems  Management  College,  Program  Manaatars 
Notebook.  Defense  Systems  Management  College,  Port  Belvoir, 
VA,  June  1992. 


89 


The  Defense  Systems  Management  College,  Systems  Engineering 
Management  Guide .  Defense  Systems  Management  College,  Fort 
Belvoir,  VA,  January  1990. 

The  Defense  Systems  Management  College,  Test  And  Evaluation 
Management  Guide .  Defense  Systems  Management  College,  Fort 
Belvoir,  VA,  March  1988. 

Department  of  Defense  Instruction  5000.2,  Defense  Acquisition 
Management  Policies  and  Procedures.  February  23,  1991. 

Department  of  Defense  Manual  5000. 2 -M,  Defense  Acqruisition 
Management  Documentation  and  Reports.  February  23,  1991. 

Department  of  the  Navy,  NAVSO  P-6071,  Best  Practices;  How  to 
Avoid  Surprises  in  the  World's  Most  Complicated  Process.  March 
1986. 

Army  Operational  Test  and  Evaluation  Agency,  Final  Test 
Report.  AAH  Operational  Test  II.  Vol.  I,  April  1982. 

Advanced  Helicopter  Improvement  Program  (AHIP) ,  Operational 
Test  II,  Report  #  TR-OT-85-774,  February  1985. 

Line  of  Sight -Forward -Heavy,  Force  Development  Test  And 
Experimentation,  Report  #  TEC-TR-90-001A/B,  December  1989. 

Line  of  Sight -Forward -Heavy,  Initial  Operational  Test  (LOS-F- 
H,  lOT) ,  Report  #  TEC-TR-90-004A,  September  1990. 


90 


INITIAL  DISTRIBUTION  LIST 


No.  Copies 


1.  Defense  Technical  Information  Center  2 

Cameron  Station 

Alexandria  VA  22304-6145 

2.  Library,  Code  052  2 

Naval  Postgraduate  School 

Monterey  CA  93943-5002 

3.  Dr.  David  V.  Lamm  2 

Code  SM/LT 

Naval  Postgraduate  School 
Monterey  CA  93943-5002 

4.  Professor  Thomas  H.  Hoivik  1 

Code  SM/HO 

Naval  Postgraduate  School 
Monterey  CA  93943-5002 

5.  Professor  William  G.  Kemple  1 

Code  OR/KE 

Naval  Postgraduate  School 
Monterey  CA  93943-5002 

6.  OASA  (RDA)  1 

ATTN:  SARD- ZAC 

103  Army  Pentagon 
Washington  D.C.  20310-0103 

7 .  Commander  1 

Test  and  Experimentation  Command 

Fort  Hood  TX  76544-5065 

8.  CPT  James  B.  Mills  2 

5878  S.  Wright  St. 

Kingsville  OH  44048 


91 


